This document was written for Daemonnews and published in DaemonNews, November 2002 issue.

Cross-Development with NetBSD
Using NetBSD's new toolchain to develop for an embedded device
Hubert Feyrer, October 2002

Introduction

When targeting a product for an embedded platform, it's not feasible to have all the development tools available on that same platform. Instead, some method of crosscompiling is usually used today. NetBSD 1.6 will contain (and NetBSD-current has today) a new framework to build both the operating system's kernel and the whole userland for either the same platform that the compiler runs on, or for a different platform, using crosscompiling. Crosscompiling requires assembler, linker, compiler etc. to be available and built for the target platform. The new build scheme will take care of creating these tools for a given platform, and make them available ready to use to do development work.

Grabbing sources

As an example, we will examine how to make a kernel for a Shark (StrongARM based system) from a machine running NetBSD/i386 in this article. The only requirement is that there is a fresh source tree of NetBSD-current checked out from CVS. "Fresh" here means that there should be no stale object files etc. in the tree - these can really get in the way here! The following commands can be used to get a copy of the -current branch checked out in /usr/cvs/src-current:

# mkdir /usr/cvs
# cd /usr/cvs
# env CVS_RSH=ssh cvs -d anoncvs@anoncvs.netbsd.org:/cvsroot co src
# mv src src-current

The process of crosscompiling a kernel consists of three steps, which we will describe in more detail below:

Create toolchain for crosscompiling (compiler, assembler, linker, ...)
Configure kernel
Build

Creating the Crosscompiler

The first step to do cross-development is to get all the necessary tools available. In NetBSD terminology, this is called the "toolchain", and it includes BSD-compatible make(1), C/C++ compilers, linker, assembler, config(8), as well as a fair number of tools that are only required when crosscompiling a full NetBSD release, which we won't cover here.

The command to create the toolchain is quite simple, using NetBSD's new src/build.sh script:

# cd /usr/cvs/src-current
# ./build.sh -m shark -u -t

This will build all the tools for the named target platform. Arguments used here are:

-m: define the machine, MACHINE
-u: update - only build files that have changed since last build
-t: only build the toolchain (i.e. the crosscompiler!)

During the build, object directories are used consistently, i.e. special directories are kept that keep the platform-specific object files and compile results. In our example, they will be kept in directories named "obj.shark" as we build for a Shark as target platform.

The toolchain itself is part of this, but as it's hosted and compiled for a i386 system, it will get placed in it's own directory indicating where to cross-build from. Here's where our crosscompiler tools are located:

miyu# pwd
/usr/cvs/src-current
miyu# ls tools/obj.shark/
tools.NetBSD-1.6-i386

So the general rule of thumb is for a given "host" and "target" system combination, the crosscompiler will be placed in the "src/tools/obj.target/tools.host" directory by default. A full list of all tools created for crosscompiling the whole NetBSD operating system includes:

miyu# ls tools/obj.shark/tools.NetBSD-1.6-i386/bin/
arm--netbsdelf-addr2line  arm--netbsdelf-strings    nbmakefs
arm--netbsdelf-ar         arm--netbsdelf-strip      nbmakeinfo
arm--netbsdelf-as         nbasn1_compile            nbmakewhatis
arm--netbsdelf-c++        nbcap_mkdb                nbmenuc
arm--netbsdelf-c++filt    nbcompile_et              nbmkdep
arm--netbsdelf-cpp        nbconfig                  nbmklocale
arm--netbsdelf-dbsym      nbcrunchgen               nbmsgc
arm--netbsdelf-g++        nbctags                   nbmtree
arm--netbsdelf-g77        nbeqn                     nbpax
arm--netbsdelf-gasp       nbgencat                  nbpic
arm--netbsdelf-gcc        nbgroff                   nbpwd_mkdb
arm--netbsdelf-gcov       nbhost-mkdep              nbrefer
arm--netbsdelf-ld         nbindxbib                 nbrpcgen
arm--netbsdelf-lint       nbinfo                    nbsoelim
arm--netbsdelf-mdsetimage nbinstall                 nbtbl
arm--netbsdelf-nm         nbinstall-info            nbtexi2dvi
arm--netbsdelf-objcopy    nblex                     nbtexindex
arm--netbsdelf-objdump    nblorder                  nbtsort
arm--netbsdelf-ranlib     nbm4                      nbuudecode
arm--netbsdelf-readelf    nbmake                    nbyacc
arm--netbsdelf-size       nbmake-shark              nbzic

As you can see, most of the tools that are available natively on NetBSD are also available, with some program prefix to identify the target platform. (The naming here is a bit redundant due to the directory structure containing all the information, but the program names are created by the GNU based toolchain and were chosen not to be changed).

One important tool that should be pointed out here is "nbmake-shark". This is a shell wrapper for a BSD compatible make(1) command that's setup to use all the right commands from the crosscompiler toolchain. Using this wrapper instead of /usr/bin/make allows crosscompiling programs that were written using the NetBSD Makefile infrastructure (see src/share/mk). We will use this make(1) wrapper in a second!

Configuring the kernel

Now that we have a working crosscompiler available, the "usual" steps for building a kernel are needed - config, then build. As the config(8) program used to create header files and Makefile for a kernel build is platform specific, we need to use the "nbconfig" program that's part of our new toolchain. That aside, the procedure is just as like compiling a "native" NetBSD kernel. Commands involved here are:

# cd /usr/cvs/src-current/sys/arch/shark/conf/
# /usr/cvs/src-current/tools/obj.shark/tools.NetBSD-1.6-i386/bin/nbconfig GENERIC

That's all. This command has created a directory "../compile/GENERIC" with a number of header files defining information about devices to compile into the kernel, a Makefile that is setup to build all the needed files for the kernel, and link them together. As the Shark port uses ELF as execution format but the Shark's OpenFirmware can only load a.out kernels, that Makefile will also convert the kernel from ELF to a.out once it's built.

More information about building NetBSD kernels can be found at http://www.netbsd.org/Documentation/kernel/.

Crosscompiling the kernel

We have all the files and tools available to crosscompile our ARM-based kernel from our Intel-based host system, so let's get to it! After changing in the directory created in the previous step, we need to use the crosscompiler toolchain's "nbmake-shark" shell wrapper, which just calls make(1) with all the necessary settings for crosscompiling for a shark:

# cd ../compile/GENERIC/
# /usr/cvs/src-current/tools/obj.shark/tools.NetBSD-1.6-i386/bin/nbmake-shark

This will churn away a bit, then spit out a kernel:

...
text    data    bss     dec     hex     filename
1687520 69632   184576  1941728 1da0e0  netbsd.aout
miyu# ls -la netbsd.aout
-rwxr-xr-x  1 root  wheel  1757216 Mar 27 02:55 netbsd.aout
miyu# file netbsd.aout
netbsd.aout:    NetBSD/arm32 demand paged executable

Now the a.out(!) kernel can either be transferred to a shark (via NFS, FTP, scp, etc.) and booted from a possible harddisk, or directly from our cross-development machine using NFS. Be sure to actually use the a.out kernel, as the Shark's firmware cannot use the one in ELF format.

Crosscompiling the whole Operating System

Of course you can not only crosscompile the NetBSD kernel, but the whole system. This is as easy as:

$ ./build.sh -m shark -d \
                -D /usr/tmp/shark-root \
                -R /usr/tmp/shark-release

This command will first build a crosscompiler, as described above. After that it will crosscompile the whole operating system including all libraries, binaries, etc. To make things complete, kernels for installing and running the systems will be built, and everything will be packed into distribution sets & install media, so a full NetBSD release is available i /usr/tmp/shark-release after that command!

Too easy? Sorry, but that's NetBSD! :-)

Summary

So much for our small example on how to use the new build framework to do cross development of a kernel for and with NetBSD. Let me re-emphasize at this point again that the toolchain produced in the first step is capable of (re)building the complete NetBSD operating system including libraries and programs, not only the kernel. More information on building the whole operating system can be found in src/BUILDING. More documentation is also available as src/tools/compat/README, which has special emphasize on setting up crosscompiling from various host operating systems, i.e. Solaris and Linux besides NetBSD.