Creating a sandbox or jail for cross-compiling

When cross-compiling dynamic libraries like GTK+ we normally have a couple of issues that complicate the compilation and later the execution of apps that depend on those libraries.
First, we have to be very careful with the includes -I and libraries -L paths since they must point to the target, otherwise we risk to overwrite the host files.
Second, after compiling we normally end up with libraries and/or apps that require the whole path that we used during the process.
For example, if we installed GTK+ in our target’s filesystem, let’s say /opt/STM/STLinux-2.3/devkit/sh4/target/usr/local, then when we execute an app in the target that is dynamically linked to GTK+, it will search the libraries in that path, instead of /usr/local.
This issue can be solved creating ugly symbolic links or modifying the .la files, but neither of them are a clean/good solution.

One method for completely eliminating these problems is to cross-compile in a protected environment in which we can install, modify and remove files without causing any damage.
Such an environment is called a jail or sandbox.

There are several ways for implementing a sandbox (like the jail package for Debian),
but the procedure shown below is quite simple and illustrates how a sandbox works.

There are two main concepts behind a sandbox: chroot and unionfs.
chroot changes the root directory of the calling process to another specified in the given path. This directory will be used for pathnames beginning with /. The new root directory is inherited by all children of the calling process.
Unionfs gives the possibility of creating a filesystem that is the union of at least two others. For example, consider that the new unionfs is the union of our host’s root filesytem mounted in read-only mode, and of a temporary filesystem mounted in read-write mode.
The result is a filesystem which looks exactly like the host’s root filesystem,
but in which you can install, modify and/or delete files without touching anything in the root filesystem.

The following procedure and script show step by step how to create the sandbox:

First we create a dedicated user that will have read/write access to the sandbox.
You can do it with any graphical interface or using the shell:

$ adduser sandbox

Give sudo permissions to this user.
WARNING: Under some circumstances this is not safe, but in our case in which we just
want to cross-compile it’s sufficient.

$ sudoedit /etc/sudoers

Add the following line at the end of the sudoers file:
    sandbox ALL=(ALL) ALL 

Set the default shell, that is the last token in a line of the /etc/passwd file, of the sandbox user to the script /bin/sandbox.sh that is explained later:

sandbox:x:1001:1001:sandbox user,,,:/home/sandbox:/bin/sandbox.sh

Now we create the temporary filesystem.

Create a 1GB file called sandbox.img filled with zeroes:

$ dd if=/dev/zero of=/opt/sandbox.img bs=1M count=1000

Create an ext3 filesystem in that file:

$ mkfs.ext3 /opt/sandbox.img

Create the empty temporary directory:

$ mkdir /opt/sandbox

Mount the ext3 formatted file filled with zeroes in the empty directory:

$ mount -o loop /opt/sandbox.img /opt/sandbox

Change it’s permissions so everybody can read/write to/from it.

$ chmod 777 /opt/sandbox

Create the directory /opt/unionfs

$ mkdir /opt/unionfs

In the /opt/unionfs directory we will mount the zeroed ext3 filesystem altogether with the host’s root filesystem taking advantage of the unionfs.
This operation takes place in the /bin/sandbox.sh script that will be executed each time
we log in as the sandbox user:

In the function mount_unionfs() in which we take one element of the mdirs list at a time, we mount the unionfs filesystem that is the union of the zeroed ext3 filesystem and the host’s root filesystem mounted in read only mode. This is the most important part of the mounting procedure.
Then we mount in the unionfs the necessary filesystem for having a working system such as /dev, /sys, /proc and /tmp
The last element is the path of the cross-toolchain that can vary.
Eg. The openmoko toolchain is located in /usr/local/openmoko/arm/bin
whilst the STM toolchain for the sh4 arch is located in /opt/STM/STLinux-2.3/devkit/sh4/bin

The chroot command is the other important part: it changes the root from / to /opt/unionfs executing the su command that contains the –shell and –login parameters.

At this point we’re logged in as the user sandbox in a safe filesystem that is the union of
the host’s filesystem and the empty ext3 filesystem. What we see in the unionfs is the host’s filesystem that was mounted in read-only mode:

$ ls /
boot dev    initrd.img  lib     media       opt  root  selinux  sys  usr  vmlinuz
bin  cdrom  etc         home    lost+found  mnt  proc  sbin     srv  tmp  var

and we can happily start compiling libraries like GTK+ in default paths such as /usr/local without risking our host filesystem integrity.

After compilation, we can log out and check out the compilation results in /opt/sandbox
We’ll see something like this:

$ ls /opt/sandbox/
gtkdfb  lost+found  usr  var

Before we logged in as the sandbox user this directory was empty (lost+found is not considered), but now it contains three new directories: the gtkdfb contains all the GTK+ sources that we compiled.
The var directory contains the cache and run with temporal information that was different than the one in the host filesystem.
The usr directory contains the local directory with all the libraries, includes, shares and everything that was generated as a result of the GTK+ compilation:

$ ls /opt/sandbox/usr/local/
bin  etc  include  lib  share  var

Now we can safely copy this directoy to our /usr/local/ of the target filesystem!
This avoids ugly logical links or other dirty solutions.

Finally, the function umount_unionfs() does the same steps as the mount_unionfs() function
but in reverse order. This is executed when the user sandbox logs out.

Here’s the script:

#!/bin/bash

# Mount a source directory or fs type $i in destination mount point $j 
# using options $k
#
#     i               j                  k
mdirs=( 
  "unionfs"   "/opt/unionfs"         "-t unionfs -o dirs=/opt/sandbox:/=ro"
  "/dev"      "/opt/unionfs/dev"     "--bind"
  "devpts"    "/opt/unionfs/dev/pts" "-t devpts"
  "shm"       "/opt/unionfs/dev/shm" "-t tmpfs"
  "sysfs"     "/opt/unionfs/sys"     "-t sysfs"
  "proc"      "/opt/unionfs/proc"    "-t proc"
  "/tmp"      "/opt/unionfs/tmp"     "--bind"
  "/dev/sda6" "/opt/unionfs/opt"     "-t ext3"
      )

mount_unionfs()
{
    mdir_size=${#mdirs[*]}

    i=0; j=1; k=2
    while [ "$i" -lt "$mdir_size" ]; do
        if [ -z "$(mount | grep -w ${mdirs[$j]})" ]; then
            echo "Mounting ${mdirs[$j]}"
            sudo /bin/mount ${mdirs[$k]} ${mdirs[$i]} ${mdirs[$j]}
        fi
        i=$((i + 3)); j=$((j + 3)); k=$((k + 3))
    done
}

umount_unionfs()
{
    mdir_size=${#mdirs[*]}

    j=$((mdir_size - 2))
    while [ "$j" -gt "0" ]; do
        if [ -n "$(mount | grep -w ${mdirs[$j]})" ]; then
            echo "Umounting ${mdirs[$j]}"
            sudo /bin/umount ${mdirs[$j]} 2> /dev/null
        fi
        j=$((j - 3))
    done
}

mount_unionfs

sudo /usr/sbin/chroot /opt/unionfs /bin/su --shell /bin/bash --login $USER

umount_unionfs

References:
chroot info page
http://www.filesystems.org/project-unionfs.html
http://www.howtoforge.com/safe_mirror_unionfs_chroot