• People owning a Marvell PCMCIA WiFi card can how have a look at the malo(4) driver. If you use this on a gumstix ARM board, have a look as some fiddling with drvctl(8) is needed to get the card recognized properly.

• Speaking about Gumstix, KIYOHARA Takashi has declared the porting effort as finished. Support for the baex, connex, verdex and verdex-pro modules is available, as is support for the support modules.

• Staying in the "embedded" corner, KIYOHARA Takashi has announced that Plathome's OpenBlockS600 (AMCC 405EX) can now boot NetBSD via NFS. See the posting for dmesg output. The OpenBlockS600 comes with a AMCC 405EX PowerPC CPU, two GigE ethernet ports and a bunch of other goodies for a price of about $600US.

• Coming from hardware to software, pkgsrc-2009Q4 was released some time ago, and of course binary packages are available for a number of platforms: 5.0/macppc, 4.0.1/sparc and 5.0.1/sparc, 4.0.1/i386, 5.0.1/i386 and the same for 4.0.1/amd64 5.0.1/amd64. Also, binaries of pkgsrc-2009Q3 are available for 5.0/shark.

• Google's Summer of Code was a big success for NetBSD and all of the Open Source community, and it seems there will be one again this summer.

  Preparations are in an early stage, but there are already a FAQ and a timeline as well as the Program Terms of Service. From the NetBSD side, we're always happy for project suggestions (please use our mailing lists for discussions), and in NetBSD, we are currently working on out projects page. If you plan to submit a proposal for a project with NetBSD, please see our project application/proposal form If you plan to submit a proposal for a project with NetBSD, please see our Project Application/Proposal HowTo.

• Getting back to the NetBSD code, a number of interesting changes were made in the previous weeks. The first to mention is that David Young has continued is work on the new shutdown order for device drivers: ``cgd, dk, dm, md, raid, and vnd gracefully detach from the device tree during shutdown. I believe that ccd is the only virtual disk that does not detach.'' This allows having arbitrary stack of file systems, and still have them unconfigured properly in the right order on system shutdown.

• Another major change that went into NetBSD recently is that terminfo was imported into NetBSD-current. Terminfo replaces termcap, but provides a backward compatible termcap interface. This move follows discussion from last summer, and docs by The Open Group (the people who make things like the POSIX standard and the Single Unix Specification), which indicates that the termcap specification will be withdrawn in the future.

• Moving from userland inside the kernel, David Holland has proceeded with work to unhook LFS from UFS. Historically, the Log structured File System was written after the Berkeley Fast File System. With the idea of sharing the core "Unix File System" code for both file systems, this resulted in a strong relationship between LFS and FFS, which was/is not always the best for the advantage and stability of either one: ``sharing ufs between both ffs and lfs has made all three entities (but particularly lfs) gross. ffs and lfs are not similar enough structurally for this sharing to really be a good design.''

• Another major addition to the NetBSD kernel was made recently by Darren Hunt: ``Courtesy of CoyotePoint Systems, I've been working on a port of DTrace [...] to NetBSD for i386.'' Citing Wikipedia, ``DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. Originally developed for Solaris, it has since been released under the free Common Development and Distribution License (CDDL) and has been ported to several other Unix-like systems.

  DTrace can be used to get a global overview of a running system, such as the amount of memory, CPU time, filesystem and network resources used by the active processes. It can also provide much more fine-grained information, such as a log of the arguments with which a specific function is being called, or a list of the processes accessing a specific file. ''

  The code is available in NetBSD-current. I haven't looked into this yet, but I'm looking forward of reports and blog postings if the wikipedia command line examples work.

  (I think like with ZFS, Dtrace could use a hand with documenting the NetBSD side of things. Any takers?)

• The last kernel change to mention is related to security: mapping the address 0 from userland was disabled. This issue went through the press late last year, and it this is now addressed in NetBSD, too. Those that still beed to map address 0 can do so via the USER_VA0_DISABLED_DEFAULT kernel option or the vm.user_va0_disable sysctl.

• So much about about the NetBSD code for now. Of course having all those fine features added screams for an immediate (*cough*) release, which brings me to the fact that NetBSD 5.0.2 has been released: `` NetBSD 5.0.2 is the second critical/security update of the NetBSD 5.0 release branch. It represents a selected subset of fixes deemed critical for security or stability reasons.

  Please note that all fixes in critical/security updates (i.e., NetBSD 5.0.1, 5.0.2, etc.) are cumulative, so the latest update contains all such fixes since the corresponding minor release. These fixes will also appear in future minor releases (i.e., NetBSD 5.1, 5.2, etc.), together with other less-critical fixes and feature enhancements. ''

• There's a hackathon going this weekend (Feb 20/21 2010)
• Volunteers are wanted to setup & man a NetBSD booth at FrOSCamp 2010 Zurich, Switzerland, on Sep 17/18 2010
• pkgsrcCon 2010 will be held in Basel, Switzerland, from May 28ths to 30ths 2010.

The wapbl(4) manpage gives more details: To enable, a kernel with "options WAPBL" needs to run, which is available in NetBSD-current since end of July 2008. Userland from a similar date is useful, as the mount(8) command needs to know about the new "log" option. With the proper system, it's pretty much a no-brainer:

1. In /etc/fstab, enable logging for the file system(s) you need, in my case it's just /:

        /dev/wd0a       /       ffs     rw,log        1 1 

   This is actually the only thing that needs to be done. All the rest writen here just explains things in a bit more details.

2. Note that journaling is not active on the file system(s) at this point, so pressing the reset button for testing will result in a file system check (fsck) - don't do it right now. :)

3. Reboot the system. Nothing special will show up in the boot messages:

        ...
        audio2 at pad0: half duplex
        boot device: wd0
        root on wd0a dumps on wd0b
        root file system type: ffs
        Fri Aug 22 20:45:55 CEST 2008
        swapctl: adding /dev/wd0b as swap device at priority 0
        Starting file system checks:
        /dev/rwd0a: file system is clean; not checking
        Setting tty flags.
        ...  

4. Let's recall what happens here: after probing the hardware and initializing device drivers (audio, ...), the kernel looks at disk drives for a file system with a root partition (i.e. a disk with BSD disklabel, "a" partition, and a known file system in it). It will use the first root file system it finds, and mount it read-only.

   As the above output is from a multi-user boot (not a single-user boot), the kernel continues to run init(8), which in turn runs /etc/rc (which then runs all of /etc/rc.d/* etc.). First things in the boot process can be determined by using the rcorder(8) tool just like /etc/rc does:

        $ cd /etc/rc.d/
        $ rcorder * | head
        wdogctl
        raidframe
        cgd
        ccd
        swap1
        fsck
        root
        ... 

   Of the above scripts, raidframe, cgd and ccd configure additional disk devices, wdogctl and swap1 are of minor interest here. The two interesting scripts are "fsck" and "root": "fsck" runs fsck(8), which in turn goes through the list of known file systems in /etc/fstab, and checks for each file system if it was unmounted cleanly last time. If not, the file system will be checked, possibly repaired, and marked as clean. This is the much-hated, time consuming process preventing a fast reboot when the system crashed.

   After ensuring all file systems are in a consistent state, the "root" script mounts the root (/) file system read-write.

   Following that, all other scripts run, create temporary files, configure network devices, enable login and whatnot. Important parts here are the order of the kernel first mounting the root file system read-only, and after checking enable writing.

5. As we have marked the root file system for journaling, the log (journal) is created when mounting the file system read-write. For NetBSD, the log has only meta-data, i.e. information on what changes were made to the file system's management data structures like directories, link counts, etc. No data blocks are journaled. This may not be 100% optimal from a user point, but it ensures that the file system is in a consistent state with respect to meta-data.

6. When the file system is mounted with journaling enabled, bad things are welcome (well, sort of :-) to happen, and the system will handle them gracefully: kernel panics, power failures, someone pressing the reset button - everything that disrupts system operation and gets the file system into an inconsistent state will be caught by replaying the journal on the next boot.

   Note that journaling will not help about user/admin errors like when you accidentally remove a file!

7. After the system went down in flames -- for research purpose and better predictability, let's assume we've pressed the reset button -- with the file system in an unclean state, this will be displayed on the next boot:

        ...
        audio2 at pad0: half duplex
        boot device: wd0
        root on wd0a dumps on wd0b
        /: replaying log to memory
        root file system type: ffs
        Fri Aug 22 20:49:55 CEST 2008
        swapctl: adding /dev/wd0b as swap device at priority 0
        Starting file system checks:
        /dev/rwd0a: file system is journaled; not checking
        /: replaying log to disk
        Setting tty flags.
        ...  

8. After finding the root file system, the kernel first recognizes the journal, and assumes that the system crashed. The system doesn't know what's up with the disk so far, so won't go and alter the disk by writing the changes from the log onto the disk. Instead, those changes are replayed to memory only. This leaves the disk as-is, but the in-memory view of the file system will be consistent.

   Running fsck then recognizes the file system as journaled, and won't touch it, assuming that the log caught all bads. Mounting the file system in the next step finally replays the changes in the journal onto the disk, and finally sets it into a consistent state permanently. After that, the regular boot process can proceed as usual.

   Please note that the messages "/: replaying log to memory/disk" are printed by the kernel, as it's the kernel that runs all the file system code.

9. When the system is up and running, the mount(8) command can be used to determine if logging is enabled or not:

        # mount
        /dev/wd0a on / type ffs (log, local) 

   The "log" here in the mount options indicates that journaling is enabled.

The enduser impact of this is that lenghty file system checks are (hopefully :-) a thing of the past now!

[Tags: ffs, wapbl]

• NetBSD achieves permanent charity status: ``The Foundation has been a 501(c)(3) charity since 2004, but previously the status was given under an advanced ruling period, i.e. it was of limited time. The permanent charity status is also known as 170(b)(1)(A)(vi).

  Being a public charity is important to us, as it means that we are eligible to receive employer matching donations, as well as to enjoy the most beneficial tax treatment. ''

• Metadata journaling support added to FFS: ``In case of a crash or unexpected power loss however, the journaled file system will not need a lengthy file system check at boot time, but instead the kernel will replay the log within seconds. This allows faster crash recovery, less overall downtime and higher availability.

  Converting an existing system to use the log feature is as easy as updating (both kernel and userland), making sure the kernel option WAPBL is selected (this is the default for GENERIC kernels now), adding a ?log? option to /etc/fstab and rebooting. Note that WAPBL is not compatible with soft-dependencies, so please ensure that you first remove the ?softdep? option if present. See the wapbl(4) manual page for more information. ''

  Kudos for this go to Wasabi Systems, Darrin B. Jewell, Simon Burge, Greg Oster, Antti Kantee, and Andrew Doran!

• Uli 'rhaen' Habel wrote me that he wrote a blosxom plugin for gnats: ``During my work for pkgsrc I started to write articles for my blog and I referred to several PRs from the NetBSD gnats system. However I just wanted to type the PR in the form of e.g. NetBSD PR pkg/39230 and would like to have my blog software to link to the webpage automatically''.

  Blosxom is the blogging software that Uli and I use, and you can learn more about his GNATS plugin, and download it, here. (Apparently I didn't get to install this plugin yet, that's why you don't see a link on the above quoted text :-).

• Stefan Schumacher wrote me that the german magazine Die Zeit has an article on operating systems showing screenshots of several operating systems, starting with C64 Basic V2, going over MS-DOS and Windows to more esoteric ones like Mac OS X, Solaris, and *cough* BSD. Check the screenshot of the latter one! ;)

• Another one from Uli Habel: His (NetBSD|pkgsrc) blog is now syndicated on www.onetbsd.org.

• Wilhelm Buehler hints me at EuroBSDcon 2008: ``EuroBSDCon is the european technical conference for people working on and with 4.4BSD based operating systems and related projects. EuroBSDCon 2008 will take place in Strasbourg, France 18-19 October 2008 at University of Strasbourg.''

• There's an article by Warren Webb titled "Free software encircles embedded design" at Electronic Design, Strategy, News (EDN). The article starts by illustriating open source software as a natural (and cheap, or course) alternative to commercial systems, describes benefits of the development model and the wealth of applications and how they can be used in an embedded environment. It continues talking about licenses, tools, and alternatives to Linux, including NetBSD.

• Those into funky gadgets may like the MoPods may be for you: ``As if a little charm pet wasn't reason enough for being, the MoPods are actually practical. When your mobile phone rings or receives a text within a metre of your MoPod then the little blighter will get in a tizz, spin round and round and a little light will flash wildly in reaction. The perfect visual warning if your phone is on silent or you are in a noisy bar.

  Whether hung on your bag, your clothes, your keys or your mobile, MoPods are a must-have, or as they say in Japan, a "hitsuyou".''

• Back to our fine operating system: Ian Hibbert, who has written NetBSD's bluetooth stack, has worked on a PAN daemon for NetBSD. This allows to perform personal area networking in various ways:

  NAP

  Network Access Point is like an ethernet bridge

  GN

  Group ad-hoc Network is a NAP with no external network

  PANU

  Personal Area Networking User in both host (like GN but a single connection) and client (the device that connects to all the others) mode.

  All this will come in an upcoming NetBSD release (well, and FreeBSD too, it seems, as they like it :-) near you pretty soon, see Iain's mail to tech-net.

So far, NetBSD has offered the Log-structured File System (LFS) for logging, but that never went beyond experimental state.

Wasabi Systems, vendor offering NetBSD-based storage solutions, has offered a version of FFS that does journaling for some time, and they have contributed their implementation to the NetBSD project now, as announced by Wasabi Systems' Simon Burge: ``Wasabi Systems Inc is pleased to make our journaling code available to the NetBSD community. This code is known as WAPBL - Write Ahead Physical Block Logging, and has been used to provide meta-data journaling in production environments for over 4 years. WAPBL journals meta-data only - not file data, and has been used on filesystems ranging from from 16MB to multiple terabytes in size.''

Please see Simon's posting for more information about the state of the code, how it works, and how to use it.

While here, one could be musing why Wasabi is contributing this to the NetBSD project at this time. The file system code this is based on is under the BSD license, so adding e.g. journaling can be kept private and there's no need to publish those changes. But software is a moving target, and so is NetBSD's file system implementation, which is undergoing great changes for fine-grained in-kernel locking and SMP right now. This means large maintenance efforts for derived code, and one could guess that Wasabi choose the step to contribute the code to the NetBSD project to get this maintenance for free. Of course the benefit to the community is that there's a journaling file system available now. Which I find is great! :-)

[Tags: ffs, journaling]