Today, I've found time to install NetBSD-current/i386, and configure things that I use here - /kern, /proc, and some NFS, in addition to a local disk. Now, looking at the list of loaded kernel modules reveals:

% modstat 
NAME            CLASS   SOURCE  REFS    SIZE    REQUIRES
compat          misc    builtin 0       -       -
coredump        misc    filesys 1       3067    -
exec_elf32      misc    filesys 0       7225    coredump
exec_script     misc    filesys 0       1187    -
ffs             vfs     boot    0       166292  -
kernfs          vfs     filesys 0       11131   -
nfs             vfs     filesys 0       145345  -
procfs          vfs     filesys 0       28068   -
ptyfs           vfs     filesys 0       8975    - 

Another interesting module is "coredump", which is loaded by the module to execure 32bit ELF programs, exec_elf32. This is an example of module dependencies, and again no manual intervention was needed.

So what modules are there? First, let's remember that kernel modules are object code that implements facilities for the running kernel, and which interfaces closely with the running kernel. As such, they need to match the kernel version, ideally. When one of the kernel's API or ABI interfaces changes, it's best to rebuild all modules. For NetBSD, the kernel's version is bumped e.g. from 5.99.15 to 5.99.16 for such an interface change, which helps tracking those changes.

Back to the question of what modules are there. Now that we know kernel modules are closely tied to the version of the kernel (which still is in the file /netbsd, btw), associated modules -- for the example of NetBSD/i386 5.99.15 -- can be found in /stand/i386/5.99.15/modules:

% cd /stand/i386/5.99.15/modules
% ls -F
accf_dataready/     drm/                lfs/                ptyfs/
accf_httpready/     efs/                mfs/                puffs/
adosfs/             exec_aout/          miniroot/           putter/
aio/                exec_elf32/         mqueue/             radeondrm/
azalia/             exec_script/        msdos/              smbfs/
cd9660/             ext2fs/             nfs/                sysvbfs/
coda/               fdesc/              nfsserver/          tmpfs/
coda5/              ffs/                nilfs/              tprof/
compat/             filecore/           ntfs/               tprof_pmi/
compat_freebsd/     fss/                null/               udf/
compat_ibcs2/       hfs/                overlay/            umap/
compat_linux/       i915drm/            portal/             union/
compat_ossaudio/    kernfs/             ppp_bsdcomp/        vnd/
compat_svr4/        ksem/               ppp_deflate/
coredump/           layerfs/            procfs/

% ls */*.kmod
accf_dataready/accf_dataready.kmod      layerfs/layerfs.kmod
accf_httpready/accf_httpready.kmod      lfs/lfs.kmod
adosfs/adosfs.kmod                      mfs/mfs.kmod
aio/aio.kmod                            miniroot/miniroot.kmod
azalia/azalia.kmod                      mqueue/mqueue.kmod
cd9660/cd9660.kmod                      msdos/msdos.kmod
coda/coda.kmod                          nfs/nfs.kmod
coda5/coda5.kmod                        nfsserver/nfsserver.kmod
compat/compat.kmod                      nilfs/nilfs.kmod
compat_freebsd/compat_freebsd.kmod      ntfs/ntfs.kmod
compat_ibcs2/compat_ibcs2.kmod          null/null.kmod
compat_linux/compat_linux.kmod          overlay/overlay.kmod
compat_ossaudio/compat_ossaudio.kmod    portal/portal.kmod
compat_svr4/compat_svr4.kmod            ppp_bsdcomp/ppp_bsdcomp.kmod
coredump/coredump.kmod                  ppp_deflate/ppp_deflate.kmod
drm/drm.kmod                            procfs/procfs.kmod
efs/efs.kmod                            ptyfs/ptyfs.kmod
exec_aout/exec_aout.kmod                puffs/puffs.kmod
exec_elf32/exec_elf32.kmod              putter/putter.kmod
exec_script/exec_script.kmod            radeondrm/radeondrm.kmod
ext2fs/ext2fs.kmod                      smbfs/smbfs.kmod
fdesc/fdesc.kmod                        sysvbfs/sysvbfs.kmod
ffs/ffs.kmod                            tmpfs/tmpfs.kmod
filecore/filecore.kmod                  tprof/tprof.kmod
fss/fss.kmod                            tprof_pmi/tprof_pmi.kmod
hfs/hfs.kmod                            udf/udf.kmod
i915drm/i915drm.kmod                    umap/umap.kmod
kernfs/kernfs.kmod                      union/union.kmod
ksem/ksem.kmod                          vnd/vnd.kmod

% find . -type f -print | wc -l
      58 

P.S.: I've seen one confusion WRT systems that use kernel modules to whatever extent, as they shrink the size of the actual kernel binary: Even with kernel modules, an operating system is still a monolithic kernel: The modules are tied in closely into the system once loaded, ending in a monolithic system. In contrast, a "microkernel" is something very different, and it doesn't have anything to do with kernel modules. :-)

[Tags: kernel, lkm, modules]

• Work on the wrstuden-revivesa is ongoing. The old Scheduler Activations (SA) based threading code that was removed from NetBSD after 4.0 is adapted for NetBSD-current, so any applications that depend on SAs can continue to run. This is important for binary compatibility.
• More changes towards the new kernel modules (kmod) framework:
  • file systems' sysctl init code is now ran in a fashion so that the modules can either be linked statically into the kernel, or loaded as module during runtime, without recompiling the code. (this used to be done via some #defines previously, which either expanded to code for the LKM, or to code for static inclusion).
  • the uaudio driver can now be compiled as kmod. More work is done to actually attach audio to newly found devices, though.
• Wasabi's journaling filesystem support was added on the simonb-wapbl branch. There are still a number of issues to be resolved before this gets to flight under real life conditions.
• Support for LVM as part of this year's Google Summer of Code was added on the haad-dm branch. Currently it is possible to create a logical volume, newfs and mount it with the Linux lvm2tools lvcreate utility - the NetBSD driver is API-compatible with Linux.
• After TNF has changed its copyright from 4-clause to 2-clause, other holders of material in NetBSD's code base have made similar changes.
• The yamt-pf42 branch was merged, which merges in a newer PF packet filter from OpenBSD 4.2.
• Management of processor sets and thread affinity was added, see the cpuset(3), affinity(3), pthread_setaffinity_np(3) and pthread_getaffinity_np(3) manpages as well as the cpuctl(8) and psrset(8) commands.
• The Red-Black-Tree code was optimized more, and moved in a place so that the same code can be used both from userland (libc) and kernel code.
• ifconfig(8) was changed to allow easy adding/removal of features such as address families (inet, inet6, iso, atalk) and protocols (802.11, 802.3ad, CARP) via the Makefile.
• SSH was extended with the HPN-SSH patch, which aims at improving performance of SCP and the underlying SSH2 protocol by dynamically allocating buffers. See the HPN-SSH homepage for more information.

SMP & audio: One area that hasn't been changed for moving towards fine-grained kernel locking was NetBSD's audio subsystem. As audio recording and playback is mostly done via interrupts, and as latency in those is critical, the audio subsystem was moved to the new interrupt handling system. The work can be found on the ad-audiomp branch, more information is available in Andrew's posting about the MP safe audio framework and drivers.

Benchmarking: Changing a system from inside out is a huge technical task. On the way, performance measurements and tuning are needed to make sure that the previous performance is still achieved while getting better performance in the desired development area. As a result, benchmarks results from Sun's libmicro benchmark suite were posted, which allow comparison not only against Linux and FreeBSD, but also between NetBSD-current and NetBSD 4.0, in order to identify if any bad effects were added. All performance tests were made on a machine with 8 CPUs, and the areas tested cover "small" (micro) areas like various system calls. Of course this doesn't lead to a 1:1 statement on how the systems will perform in a real-life scenario like e.g. in a database performance test, but it still help identifying problems and gives better hints where tuning can be done.

Another benchmark that was also made in that regard comes from Gregory McGarry, who has published performance measurements previously. This time, Gregory has run the lmbench 3.0 benchmark on recent NetBSD and FreeBSD systems as well as a number of previous NetBSD releases - useful for identifying performance degradation, too!

One other benchmark on dispatch latency run was made by Andrew Doran: on a machine that was (CPU-wise) loaded by some compile jobs, he started a two threads on a CPU that wasn't distracted by device interrupts, and measured how fast the scheduler reacted when one thread woke up the other one. The resulting graph shows that the scheduler handles the majority of requests in less than 10us - good enough for some realtime applications?

Kernel modules are another area that's under heavy change right now, and after recent changes to load modules from the bootloader and the kernel, the kernel build process was now changed so that pre-built kernel modules can be linked into a new kernel binary, resulting in a non-modular kernel. Eventually, this could mean that src/sys is built into separate modules, and that the (many) existing kernels that are present for each individual platform -- GENERIC, INSTALL is already gone, ALL, etc. etc. -- can be simply linked from pre-compiled modules, without recompiling things over again for each kernel. Of course the overal goal here is to speed up the system (and kernel!) build time, while maintaining maximum flexibility between modules and non-modular kernels.

With the progress in kernel modules, it is a question of time when the new kernel module handling supercedes the existing loadable kernel modules to such an extent that the latter will be completely removed from the system -- at least the latter was alredy proposed, but I'd prefer to see some documentation of the new system first. We'll see what comes first! (Documentation writers are always welcome! :-)

[Tags: benchmark, kernel, smp]

Changes related to SMP:

• Yamamoto Takashi has started the yamt-nfs-mp branch to make the NFS client MP-safe
• After merge of the yamt-lazymbuf branch, the send(2) and recv(2) system calls are MP-safe
• Other system calls that have been made MP-safe are for NTP, PMC, reboot, sysarch and time. With the exception of the Darwin and Irix emulations, all system falls are now MP-safe!
• Progress on the wrstuden-revivesa branch to get back support for Scheduler Activations. Much of the code that was removed when Andrew's 1:1 threading was added is put back in a way that both threading mechanisms can co-exist. Affected areas are the interface to the generic scheduler and locking.

• In the networking code, stats for ICMP, ICMP6, UDP, TCP, IP and IPv6 were changed from a structure to an array of uint64_t values by Jason Thorpe. This removes a few structs from the kernel header files. The change is ABI compatible with the old structures, as such tools like netstat(1) will continue to work.
• Also, while moving towards a multi-threaded network stack, stats for protocols like UDP6, IP, PIM6, ARP, IGMP, IPSEC, IPSEC_FAST, PF_KEY, Appletalk DDP, and CARP are accounted on a per-cpu base, and routines were added to support collating per-cpu-gathered network statistics.
• ifconfig(8) got a major overhaul towards improved modularity and extensibility. The internal parser's cleaner, and it should be easier to add new commands.
• In the search for replacing the ISC DHCP client dhclient(8) with something smaller, Rob Marpled's DHCP Client Daemon dhcpcd(8) was imported. It is 1/6 of the size, yet has about all the features plus adds support for more modern RFCs like IPv4LL (RFC 3927), Classless Static Routes (RFC 3442) and Node-specific Client Identifiers (RFC 4361).
• Kernel support for adding/removing link-layer (i.e. MAC/ethernet) addresses using SIOCALIFADDR AND SIOCDLIFADDR, respectively. Corresponding ifconfig(8) changes were announced to come soon.

• Progress on the mjf-devfs2 branch: adding wedge support, devfsd is started by init(8) before going multiuser
• Thor Lancelot Simons has extended the crypto(4) interface to handle asynchronous operations. Ioctl calls to create, submit/retrieve and destroy multiple sessions were added, which should make it easier to write new crypto applications overall. The code for this was contributed to TNF by Coyote Point Systems, Inc.
• Clauses 3 and 4 were removed from TNF licenses. There's an outstanding press release on this one, but interested parties are OK to remove those closes from existing older code, following UCB's prior example.
• Kernel-option MULTIPROCESSOR is now mandatory on i386
• i386 and amd64 now have a default boot.cfg file, allowing to boot either single user or multi user (= normally), wither with or without ACPI and/or SMP.
• For new kernel modules, the suffix will be .kmod (over the old .o, see your /etc/rc.lkm.conf), and the designated place is /kernel/modules for now. A number of drivers that aren't in GENERIC are now built as modules to allow easier testing or loading when needed, and the x86 boot loader can now load new style modules and pass them to the kernel. Also, the miniroot (ramdisk) for kernels can be loaded as a module as well - this will mean one kernel for both installation and running - no more separate INSTALL kernels, yai!

  Curious parties can try the new scheme by setting MKMODULAR=yes in their /etc/mk.conf file and by adding option MODULAR to the kernel config, see src/share/mk/bsd.README for more information.

• The UDF DVD/etc. file system now supports writing. Use the new mmcformat(8) tool to format rewritable CD/DVD discs and newfs_udf(8) to create the filesystem.
• Support for the LC_MESSAGE, LC_MONETARY and LC_NUMERIC locale categories was added.
• src/external was made as new directory where "external" sources from 3rd party projects will live in the future. This includes the sources that are in src/gnu/dist, src/dist etc. right now. Src/external has subdirectories named by license, and src/gnu/dist will become src/external/gpl (or so) in the future, OpenLDAP was the first package to be imported here, it lives in src/external/bsd/openldap.
• Numerous drivers were added:
  • isv(4) for the IDEC Supervision/16 ISA image capture board
  • finsio(4) for the Hardware Monitor in the Fintek LPC Super I/O chips
  • amdtemp(4) for AMD CPU Temperature Sensors
  • hpqlb(4) for the HP Quick Launch buttons on the HP Pavilion notebooks
  • acpidalb(4) for PNP0C32 Hotkeys AKA "direct application launch buttons"
  • cpi(4) for the Creative Systems Inc. Hurdler CPI parallel printer card
  • siisata(4) for the Silicon Image SteelVine SATA-II controllers
  • lii(4) for the Atheros/Attansic L2 Fast-Ethernet chip found e.g. on the Asus EeePC
  • uberry(4) to charge a RIM BlackBerry on a USB port
• Various imports and updates of in-tree 3rd party software:
  • IPfilter was updated to 4.1.29
  • libevent was updated to 1.4.4-stable
  • OpenSSL was updated to a snapshot from 20080509
  • OpenSSH 5.0 was imported
  • ATF 0.5 was imported
  • libarchive 2.5.4b was imported
  • nvi 1.81.6 was imported
  • OpenLDAP 2.4.9 was imported in the new src/external directory
  • nawk was imported from the 20070501 Bell Labs sources
  • + probably others that I've missed

• Kernel Preemption: Andrew Doran has continued his work towards fine-grained locking, and he has proposed a patch to implement kernel preemption, i.e. that in a realtime environment, high-priority processes can interrupt system calls running inside the kernel.

  Handling the Floating Point Unit (FPU) was added later on -- the FPU needs special attention as saving and restoring is expensive, and doesn't need to be done in many cases. But if a program uses it, care must be taken to handle the case. The exact handling is explained by Christoph Egger.

  While there, Christoph also outlined the roadmap for getting realtime support in NetBSD - there are still a number of bits missing, but being able to preempt the kernel is a good first step!

• Fine-grained socket locking: In order to allow fine-grained locking (instead of blocking all other processes from entering the kernel, as is done in the "biglock" SMP approach), many kernel subsystems need to be changed. The socket system is the core part of interprocess communication, and Andrew Doran has changed it to use fine-grained locking now.

  In that context, the question of what code still runs with the biglock held, and Andrew gave an overview where more work is needed: some file systems (lfs, ext2fs, nfs), most of the drivers, protocols like TCP/IP, Veriexec, and some machine-dependent parts.

  Veriexec-Hacker Brett Lymn added details on the status of Veriexec with respect to its transition towards fine-grained locking.

• Kernel modules and ramdisk: A change in kernel modules was proposed some time ago, and Andrew Doran has used this scheme now to unify the way many ports handle the install media: There, the kernel loaded contains a ramdisk (miniroot) image inside the kernel, which is then used as root-filesystem for the kernel, containing the install tools.

  In order to split things and eventually use a stock GENERIC kernel for both running and installing, Andy has changed the x86 boot process to load the miniroot as a kernel module.

  When booting it may be useful to select one of several ramdisks: one for installing, and one for resuing the system, For this, the recently introduced boot.cfg file was extended to handle kernel modules in the boot menu.

  Izumi Tsutsui has made an ISO with all changes for testing available.

• Device File System (devfs): Another area of the kernel where a lot of work is currently being done by Matt Fleming is NetBSD's device driver infrastructure, esp. under aspects of dynamic attaching, detaching, and suspending (power management!). To talk to the various drivers, device nodes in the /dev directory are kept right now, but those are static and need to be updated when a new driver is added. Matt is working on a Device Filesystem (devfs) that dynamically created /dev from the list of devices inside the kernel. The fileysstem will also handle dynamic creation and deletion of nodes, and as an important case it will also keep permissions across reboots, if someone changes permissions manually.

  The work is at a very mature point right now and needs some testing - see Matt's mail to the tech-kern list for more information!

• Testing driver attachment: While talking about testing of device drivers, David Young has reminded driver developers to test individual drivers' detachment and re-attachment, suspension and resumption after changes. He has also posted a how-to for those tests, using drvctl(8). (The manpage needs some updating, sorry -- UTSL :-)

Another project that was started by Antti after his work to run filesystem code in userland is "rump". The project allows to use "ordinary" filesystems that usually run inside the kernel, and mimic an environment similar to what's available inside the kernel, and move the whole filesystem into userland - verbatime, with no code changes! This allows to develop filesystem code in userland, and later on move it inside the kernel with no further changes - a bit step forward for filesystem development!

This all sounds rather easy, but as filesystems need to move data between storage and memory, a big issue in filesystems is interfacing with the virtual memory subsystem, and adding interfaces like puffs and ReFUSE also needs to consider VM for efficient transfers and caching.

Work in this area is still ongoing, and I've asked Antti about his recent achievements in this area[1]. While the only user-visible change is caching and performance improvements in the Secure Shell filesystem's handler "mount_psshfs", most of the changes are on the inside. Antti wrote me: ``The interesting ones from a programmer's perspective are probably:

• Splitting userspace transport out of puffs in the kernel (putter)
• Using putter to implement support for userspace block/char device drivers (pud). pud does still not have a userspace library similar to libpuffs. libpuffs needs to become libputter and lib{puffs,pud}.

• Removing special case handling for the puffs user/kernel protocol transport. This means that file system requests can now be read/written like any other protocol. This is covered in the AsiaBSDCon 2008 paper "Send and Receive of File System Protocols: Userspace Approach With puffs"

  With some minor work in libpuffs, it possible to e.g. do an ffs mount from a remote site with the help of rump.

With puffs and rump, there are two very interesting and active projects doing research in filesystems on NetBSD, which may lead to changes in the way filesystems are understood in the Unix world. While there, a third project that may be worth watching in this regards is this year's Google hurdt Summer of Code project by Marek Dopiera, which aims at implementing Hurd translators for NetBSD,

Update: Antti dropped me a note that another project related to filesystems is this year's "fs-utils" SoC project. The goal is to create a userland tool to manipulate filesystem images, and the idea is to reuse kernel code with the ukfs library. That way, no redundancy between kernel sources and userland sources are created, and both areas benefit from mutual testing and code maturity.

[Tags: filesystem, fuse, google-soc, kernel, puffs, refuse]

• an in-kernel loader/linker so there's no need to rely on running the ld(1), and rely on have a working userland, thus
• module dependencies so that one module can request to load other modules automatically
• support to load modules from the boot loader and provide them to the kernel
• use the same code for kernel modules and in-kernel use, so things that are currently used inside the kernel can be moved to a module easily, without changes in code.

A possible solution is available in OpenBSD's config(8) command: By running "config -e /kernel", userconf commands can be "saved" into the kernel binary, preventing the need to re-run userconf on every boot.

Jared McNeill has proposed another approach for NetBSD now: Instead of modifying the kernel binary, have the bootloader read a list of (userconf) commands, and have the kernel execute them automatically.

Instead of introducing yet another config file format, Jared has opted for (re)using the proplib API functions to load the config file from disk and pass it on to the kernel. Those crying "YEEK, XML!" now can rest assured: there's a policy in NetBSD that XML is not used for config files that the user needs to edit, and the idea is to use userconf as usual, then dump the settings to the config file and use that on the next boot, see Jared's second patch for the most recent code version.

With this scheme, there's a common file where boot-time information can be stored, and the eventual idea is not only to have all ports' bootloaders read that file, but also store further information into the file to make settings other than those available via userconf today: Jared's ideas include storing bootloader settings (timeout, serial console speed, ...) and kernel tuneables like PCI_*_FIXUPs in there. I guess we can stay tuned to see what will happen on this front!

• A new set of synchronization primitives in the kernel designed to make programming for multiprocessor systems easier and more efficient: mutexes, reader / writer locks, condition variables, sleep queues and MI memory barrier operations.
• A number of underlying kernel facilties have been made 'multiprocessor safe' including the scheduler, ktrace and the general purpose method of kernel synchronisation: sleep & wakeup
• Some application facilities have been made MP safe and can now run without the "big lock" on multiprocessor systems, including signalling, SysV messaging, and system calls that inspect process state, for example: wait().
• The number of system calls that will run without the big lock went from 1 up to 56, with more in the pipeline. For workloads that are fork intensive and make heavy use of signals this will show a small yet quantifiable benefit on multi-way systems.
• The branch introduces a new 1:1 threading model that allows multithreaded applications to take advantage of all available CPUs in a multi-way system. The scheduler activations implementation used from NetBSD 2.0 through NetBSD 4.0 provides execellent performance on single CPU systems, but restricts any instance of a threaded application to a single CPU in the system. Given that multicore and multi-CPU systems are increasingly commonplace and that single threaded CPUs are rapidly disappearing from the market, we made the decision to move to a new threading model, on the basis that providing increased concurrency is now the most important factor in ensuring good performance for threaded workloads.
• Those following source-changes already know what that new 1:1 threading model means for the scheduler-activations based m:n threading model: it's gone.

(If someone wants to include Jochen's text into the NetBSD Internals Guide, that'd be great... just like any other work in that area. Any takers? Send your patches to netbsd-docs@, feel free to CC: me!)

[Tags: Docs, kernel, openbsd]