• NetBSD's "let's move kernel parts to the userland" RUMP project is still under heavy development, and in order to make testing of compatibility after kernel changes easier, a new command "rumptest" was added to build.sh: ``Basically you say:

  	    ./build.sh ${yourargs} tools ; ./build.sh ${yourargs} rumptest
  	

  Where yourargs are what have you, e.g. '-U -u -o -O /objs'.

  The latter builds only the rump kernel libs and uses some ld+awk magic to figure out if things go right or not. This is to avoid having to install headers and build libs (which is too slow since a full build is too slow). The magic is not a substitute for a full build, but it is n+1 times faster and works probably 99.9% of the time.

  The scheme uses a number of predefined component sets (e.g. tmpfs+vfs+rumpkern) to test linkage. They are currently listed in build.sh. This area probably needs some work in the future. It would be nice to autogenerate the combinations somehow.

  If things go well, you get something like this:

          ===> Rump build&link tests successful
          ===> build.sh ended:   Wed Nov 18 20:10:59 EET 2009 

  '' See Antti's Antti's mail to tech-kern: on how to tell if things didn't go so well, and what to do in that case.

• According to Wikipedia, ``Non-Uniform Memory Access or Non-Uniform Memory Architecture (NUMA) is a computer memory design used in multiprocessors, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.''

  Supporting NUMA in a contemporary (i.e.: Intel centric) SMP-enabled operating system requires following a bunch of standards, two of which are parsing of two tables, the System Resource Affinity Table (SRAT) and the System Locality Information Table (SLIT). Both tables are accessible via the Advanced Configuration and Power Interface (ACPI), and according to the German-language Wikipedia, the SRAT is used to assign local memory to local threads to boost their performance, and the SLIT defines the "distance" of the nodes among themselves, which is used to determine the "nearest" memory if local memory is not enough.

  Now, Christop Egger has posted patches to add an ACPI SLIT parser and an ACPI SRAT parser. See the two postings for dmesg pr0n from his tests on an 8-node system.

• Staying with ACPI and Christoph Egger, he found that even though the ACPI spec defines an ACPI device for fans, BIOS vendors and OEMs do their own thing. To accommodate things like the fan sensor found in the ACPI Thermal Zone in his HP Pavillion DV9700 laptop he has proposed a driver to extend the acpitz(4) driver with fan information. That way, envstat(8) can be used to display the ran's RPMs:

  [acpitz0]
    Processor Thermal Zone:     56.000   95.000                       degC
                       fan:       2840                                 RPM 

• Staying with driver games, iMil writes me that there's documentation on getting DRI, AIGLX, Composite and Compiz going with NetBSD 5.0 available in the O(ther)NetBSD Wiki now.

  The documentation covers how to enable the Direct Rendering Manager (DRI), setting up and configuring Modular X.org, assuring that everything's in place, and how to get Compitz going. Mmm, wobbly windows at last! :-)

• While we're talking funky desktop stuff: Marc Balmer has submitted a patch to get touchpanel support for ums(4). ums(4) is for USB mice, and in contrast to mice, touch panels need to deal with absolute numbers, not relative numbers.

• Back to the guts of the kernel, another patch suggested by Christop Egger was for adding x2apic. What is x2apic? X2APIC is ``an Intel-only feature but can also be found in virtual environments with support for CPU apic id's > 0xff.

  I.e. Xen 4.0 (not yet released) supports 128 CPUs in HVM guests with the CPUs enumerated with even apic id's. That means you need x2apic for the 128th CPU :) ''

• While speaking of Xen: Xen 4.0 is coming soon, and there's a call to help testing it on NetBSD!

  Install Mercurial, check out latest Xen sources, apply a bunch of patches, build and install. Examples of commands are given, in addition to changes required for /boot.cfg etc.

  Report your findings to port-xen!

• Last one for today: Michal Gladecki, Editor-in-Chief of BSD Magazine writes: ``We are happy to announce that BSD Magazine is transforming into a free monthly online publication. The online version of BSD Magazine will stay in the same quality and form. It will look like the BSD magazine one is familiar and comfortable with. Please sign up to our newsletter at www.bsdmag.org and get every issue straight to your inbox. Also, you can now download any of the previous issues from our website. The first online issue -- 2/2010 -- is coming out in February. Please spread the word about BSD Magazine. '' Click!

Unrelated, I've been playing with git a bit over the past few days, and wile I have a number of questions building up (which will be subject to tech-repository or so), what I can say today is that the speed of "git pull" with NetBSD's git repository and my 1MBit DSL line reminds me a lot of the times when I used SUP with my 56k modem - it took forever, too. :-(

[Tags: acpi, acpitz, aiglx, Articles, bsdmac, compiz, dmesg, dri, numa, rump, slit, touchpanel, ums, x2apic, xen]

With RUMP, PUFFS and the like, Antti is set on a mission to define interfaces between kernel components, and move those kernel components from inside the kernel to the userland. As a result, the (once) kernel-code is now running as "simple" userland processes that have all the benefits like memory protection and not taking down the whole system in case of an error. Examples that Antti has been bashing on started with moving file systems to the userland, which resulted in PUFFS and ReFUSE, an implementation of the Linux inferface of File systems in USErland (FUSE). Not stopping at file systems (which are traditionally used for accessing data on storage media, like e.g. for ntfs-3g), Antti went further and moved into getting network services like SSHFS going - in userland, based on FUSE.

Not stopping there, more recent works include:

• USB device driver support, to run NetBSD's USB stack in userland - and with it a bunch of USB device drivers as well, e.g. storage (umass) and wireless adapters.

• Running NFS in userspace, and with it the underlying TCP/IP stack? Well, we've had nfsd(8) for quite some time, but this time it's the same code running in userland as in kernel. The thread also answers questions on why we want to virtualize nfsd+networking with a process and why do we want to do that by reusing kernel code.

Sounds impossible? Ye fear not, it's been done! Arnaud 'stacktic' Ysmal already has ported RUMP to FreeBSD and Linux, the work is available via pkgsrc/misc/rump, and there is also Arnaud's page on Rump on non-NetBSD Operating Systems for more information.

Whew... lots of research and development going on in this area, and - getting back to the initial topic - we may well see an operating system in the future that moves from the monolithic to a microkernel approach, and it may or may not be called NetBSD. Fact is, that a lot of research is going on in that area, on NetBSD, here and now. Hats off, Antti!

[Tags: fuse, microkernel, nfs, ntfs-3f, puffs, refuse, rump, sshfs, tcp/ip, ukfs, usb]

• Web-based firewall log reporting and analysis tool Webfwlog 0.94 released: ``Webfwlog is a flexible web-based firewall log analyzer and reporting tool. It supports standard system logs for linux, FreeBSD, OpenBSD, NetBSD, Solaris, Irix, OS X, etc. as well as Windows XP. Supported log file formats are netfilter, ipfilter, ipfw, ipchains and Windows XP. [...]

  With Webfwlog you can design reports to use on your logged data in whatever configuration you desire. Included are example reports as a starting point. You can sort a report with a single click, "drill-down" on the reports all the way to the packet level, and save your reports for later use. You can also create a link directly to any saved report.''

  See the webfwlog homepage for more information.

• Git copies of cvs modules available - spz@ writes that ``htdocs, othersrc, pkgsrc, src and xsrc are now available as (bare) git repository copies at http://ftp.netbsd.org/pub/NetBSD/misc/repositories/git. These repositories are currently updated every 30 minutes.

  They are not authoritative and are not meant to replace cvs, but if you want to track your development that is not yet fit for committing into cvs head or sending in as a PR, and want to use git for that, they may serve as a convenient starting point or reference.''

• Support for writing file systems on sparse disk images - File system hacker Antti Kantee writes: ``Yesterday I wanted to write to a file system which was on a sparse disk image. Normally I would have just used a rump mount, but since the file system was not at offset = 0 in the file, this failed. vnd wasn't helpful either: it allowed me to mount the image and then I got weird errors when writing.

  I just finished adding disklabel support to the various rump_fs utilities. Due to them using the option parsing code of the real mount_fs utilities, I decided to signal the label number with a magic parameter at the end of the device (yes, this really simplifies things currently).

  Let's assume you're looking at wd0.img. Here's how it works:

  === SNIP ===
  golem> disklabel wd0.img
  [...]
  16 partitions:
  #        size    offset     fstype [fsize bsize cpg/sgs]
   a:    719712        63     4.2BSD   1024  8192     0  # (Cyl.      0*-    714*)
   b:     66657    719775       swap                     # (Cyl.    714*-    780*)
   c:    786369        63     unused      0     0        # (Cyl.      0*-    780*)
   d:    786432         0     unused      0     0        # (Cyl.      0 -    780*)
  
  golem> rump_ffs wd0.img%PART:a% /puffs
  rump_ffs: "wd0.img" is a non-resolved or relative path.
  rump_ffs: using "/usr/home/pooka/wd0.img" instead.
  golem> df /puffs
  Filesystem                      1K-blocks       Used      Avail %Cap Mounted on
  /usr/home/pooka/wd0.img%PART:a%     338471     256211      65337  79% /puffs
  === SNIP ===

  So you specify the partition number as %PART:n%, where is n is obviously the partition number. Other than that, things work as usual. No vnconfig etc. necessary. ''

• New ACPI driver: acpismbus(4) - call for testers - Paul Goyette writes that ``At the request of some folks off-list, I have created a driver for ACPI's SMBus Control Method Interface. Basically, this is an acpi-based wrapper to provide device-independant access to i2c/SMBus controllers. (See spec at http://smbus.org/specs/smbus_cmi10.pdf for further info.)

  To see if your system can use this driver, check the output of acpidump for the existence of a Device(SMB0). If you have an SMB device, I'd appreciate it if you could help test this driver!''

  Continue with Paul's mail to learn on how to test the driver, and what limitations there are currently.

• Google Summer of Code is over for some time, but apparantly no final report has emerged so far (shame!). Still, a number of individual status reports came by on the official lists:
  • Zach Wegner's sysinst project
  • Adam Burkepile's subfile project (PDF)
  • Martin Schütte's syslogd project
  • Jeremy Morse's DVB project
  • Mark Dopiera's hurd translators project
  • Adam Hamsik's lvm project

  I know of at least one other project (uvc) that has completed but that I didn't see a report here - maybe I've missed it. Anyways, GSoC was another big success this year. Thanks, Google!

• Speaking of Adam Hamsik and Logical Volume Management (LVM), Adam has continued his work in that are, and he has written a device mapping library that interacts with his kernel driver. This allows to interact with his GSoC project without using any GPL code! See Adam's posting for more details.

• Force 10 Networks, producer of 10gbit switches that use an operating system based on NetBSD, have added a new feature as part of their FTOS operating system: VirtualView, which provides virtualization of Force 10 based equipment. From the xchange article: ``Force10 Networks Inc. this week introduced VirtualView software for benchmaking, troubleshooting and managing virtualized environments based on Force10 gear.''

  More information is available from the Force 10 Networks homepage, plus in articles by fibresystems.org, light reading, Zycko, and SmartBrief.

• Following the latest hype in portable computers, NetBSD has created a netbook page that intends to list models and the extent to which they are supported. Your contributions are most welcome here! (Contact me for sending updates and hardware :-)

• Zafer Aydogan has made RSS feeds available for CVS commits to single files - see his mail to netbsd-users for more details.

• New security advisory were released that I've missed in my last update:
  • NetBSD Security Advisory 2008-011: ICMPv6 MLD query
  • NetBSD Security Advisory 2008-012: Denial of service issues in racoon(8)

• A project that's been ongoing for quite some time is the move from "old-school" loadable kernel modules (LKMs) to new-style kernel modules. Important changes include the fact that modules can be either linked into the kernel at build time, or loaded into the kernel at runtime from the same file. Also, the bootloader was modified to load modules after the kernel, e.g. for a RAM-disk like the one that is used by the INSTALL kernel.

  In the same line, some parts are starting to be moved out of the GENERIC kernel, and installed as modules that can be loaded by the new framework then. The start is made by POSIX semaphores as a first step and proof-of concept, even if some details are still under hot debate, e.g. what the file system layout for modules is, and if the belong to the kernel and its build process, or to the userland.

• While talking about splitting the kernel into modules, Antti Kantee has continued his work to move parts of the kernel into userspace, in particular running file system code as userland in his RUMP, and puffs and (Re)FUSE works. The idea is to provide the interfaces that file systems need in the userland, and the result is that you can run code that used to run inside the kernel in userland now.

  Another subsystem running in the kernel that could be moved to userland by providing appropriate interfaces with the rest of the kernel is the network stack, and Antti has moved just that to the userland. See Antti's mail to tech-net@ for more information on this impressive work.

• NetBSD has shipped XFree in previous releases, and people who wanted to use X.org had to install it from pkgsrc. That's all fine, but to get a modern X, one had to compile things, as no precompiled binary packages are made available for many platforms. This is changing now, and NetBSD is getting X.org integrated via a reachover infrastructure which is also enabled for crosscompiling.

  The "user interface" for this is still in flux, but after some detour ("build.sh -V MKXORG=yes", without -x), "build.sh -x" now builds whatever X is considered the default for the platform. Some platforms already default to use X.org as X, and more will come, as changes that were made to NetBSD's copy of XFree are adopted to X.org.

  Platforms that use X.org by default now are macppc (see here and here), sparc sparc64, shark, amd64 and i386.

  As X.org is at Revision 7 now, it's installed in /usr/X11R7, which will lead to a lot of interesting effects. pkgsrc is already prepared for the new layout, but there are still many minor details that will need adjusting to the new directory. If you find one, post your patches to tech-x11.

• Besides the GNU C compiler, there's the BSD-licensed Portable C Compiler around for some time now. It doesn't offer the same support as its GNU cousin yet, but this may change now: The BSD Fund is currently doing a fund drive to get money to enhance PCC. The goal is to raise $12,000US to improve support for core compiler functionality as well as support for C99, gcc compatibility and the amd64 architecture. See the project page for further details.

• The NetBSD 5.0 release cycle has started! There's a netbsd-5 branch in CVS, daily binaries are available for testing, and some of the highlights of the upcoming release include file system journalling for FFS via WAPBL, and X.org.

  To help testing of NetBSD on Cobalt machines, Izumi Tsutsui has made a NetBSD 5.0_BETA based version of the Cobalt restore CD available. Enjoy!

• As the final point today, a word on NUMA support from Christoph Egger. Non-Uniform Memory Access is needed in massive parallel systems where some nodes have RAM more tightly associated than others, where the RAM is further away, resulting in different access times for different regions of memory. In order to support this, Christop Egger has made first steps.

  His example implementation uses information from ACPI, and shows some heavy dmesg-pr0n from a 16-core machine with four sockets. Yumm!

• wscons: Expansion for wstablet in NetBSD
  Student: Jason W. Beaudoin

  To support tables for the wscons console driver, a number of changes to the wscons API have been proposed in the student's status report. While there are a number of similarities with the wsmouse interface, there are also a number of differences that need to be worked around, e.g. absolute vs. relative coordinates. Many of the proposed changes are implemented, and we're looking forward to complete this project successfully within time.

  Status report, project page, project proposal

• subfiles: Subfile Support for NetBSD
  Student: Adam Burkepile

  Subfile allow to associate data with a regular file, just like regular allow to associate data with a directory. New internal data structures were defined to identify subfiles within the file system, and tools like newfs and dumpfs were adjusted. An API is being designed to access subfiles, and work to realize the assorted functions is being approached.

  Status report part 1 / 2, project page, project proposal

• hurdt: Hurd translators
  Student: Marek Dopiera

  Translators are programs which provide filesystems in user space functionality. This is provided via NetBSD's RUMP interface, and additional system calls and file system operations have been defined to activate the server processes when access to such a "translator" is made. Currently, the translators are only implemented in NetBSD's ext2 file system, as this allows testing of interaction with Hurd - Hurd's support for FFS seems non-working, unfortunately. The project's under busy development, and we're looking forward to see the final results.

  Status report, project page, project proposal

• lvm: Write and improve NetBSD LVM driver
  Student: Adam Hamsik

  This project implements the Linux LVM API (libdevmapper) on NetBSD, to allow using Linux' lvm2tools for logical volume management. The project's making excellent progress, there is an ISO image (see URL in status report) as well as a qemu image available for testing, and latest reports show that linear volumes can be configured and mounted already.

  Status report, project page, project proposal

• uvc: Add support for UVC devices (USB web-cams)
  Student: Patrick Mahoney

  A kernel driver was developed to read data from webcams using a custom API, work to implement the video2linux API is under way. Current challenges lie in NetBSD's USB stack, which lacks support for isochronous transfers, which is used by many (but not all) webcams. The project has made excellent progress so far, and we're positive that the project will be a success.

  Status report, project page, project proposal

• dvb: DVB drivers and kernel framework
  Student: Jeremy Morse

  This project implements a driver for Digital Video Broadcasting to supplement bktr(4)'s TV card support. So far, a driver and an API to transport data from the kernel to userland was implemented, matching LinuxTV. The project's making good progress, even due to conflicts with the academic schedule.

  Status report, project page, project proposal

• install-tool: Customizable install tool for NetBSD
  Student: Zachary Wegner

  NetBSD's current installer, sysinst, is being split into frontend and backend parts, with a configuration file building the interface between the two parts. Untangling the current mix of user interaction and install operation are ongoing, with challenges like request of install media (think floppy #42) and network configuration. Also, a parser for the configuration file was written, and work not started yet is the frontend creating the configuration file for the backend.

  Status report, project page, project proposal

• fs-utils: File system access utilities
  Student: Ysmal Arnaud

  This project is using NetBSD's RUMP and the ukfs library to access a file system image from userland. So far, makefs(1) can generate a file system image, and it can now be manipulated as well. Both a "file system console" as shell to operate on the image as well as separate tools for single operations have been designed. Many of the "normal" userland tools' functionality like ls(1), cp(1) and rm(1) were implemented. This project has made excellent progress so far, see the status report and project page.

  Status report, project page, project proposal

• cwrapper: pkgsrc: rewrite wrapper framework in C
  Student: Amitai Schlair

  The student was distracted by travel obligations for some time, but work has started in pkgsrc now. Current work includes a set of ATF tests to cover the usage of the existing pkgsrc wrapper framework and a design plan for the new wrapper implementation. The actual wrapper implementation remains to be written, and we're looking forward to see the it happen, including integration into pkgsrc plus benchmarks on the increase in speed.

  Status report, project page, project proposal

• atfify: Converting remaining regression tests to the Automatic Testing Framework
  Student: Lukasz Strzygowski

  The Automated Testing Framework was added to NetBSD as a result of last year's Summer of Code. This year, all the remaining regression tests from src/regress are being converted to ATF. Test suites for tools (awk, grep, make, ...) and libraries (libm, libpthread, ...) were converted so far. Tests for libc are currently being converted, and kernel tests are next. We're looking forward to get all of src/regress changed to ATF!

  Status report, project page, project proposal

• ext3: Implement Ext3 file system support
  Student: Rus-Rebreanu Alin-Florin

  This project intends to implement journaling in the file system by reusing Wasbi's wapbl code. Unfortunately, little has happend to reach this goal (to say the least), and the student got AWOL. Interested parties are welcome to try out ext2fuse (which also does ext3, despite the name) for now.

  Project page, project proposal

• packet-classes: Create an in-kernel API for "packet classes"
  Student: Anish Babu

  Nothing has happened in this project as well. I hear about communication problems, and it remains to see if things move forward here. :-(

  Project page, project proposal

• teredo: Implementation of RFC4380 (Teredo) in NetBSD
  Student: Arnaud Lacombe

  The project aims at creating a Teredo client, server and relay via a kernel pseudo device and assorted userland tools. The project is currently still in the analysis and design phase, which will give it little time for completion -- as for other students, this is due to a clash with the academic schedule of the student. We're holding up hopes that the set goals can be met at least partially.

  Project page, project proposal

• syslogd: Improve syslogd
  Student: Martin Schütte

  The project aims at implementing the upcoming IETF standards for syslog transport over TLS (instead of UDP), a more formal format of the messages themselv for easier automatic parsing, and signing of messages to assert authentication, integrity and correct sequencing of syslog messages. The first part of this is already working, the second part is development. We're looking forward to see this project succeed, as it will be of benefit not only to NetBSD.

  Project page, project proposal

Update: Fixed a typo (thanks tron!), got an update on the ext3 project, and added a link to the status report of the cwrapper project, which I've missed (sorry!).

[Tags: atf, dvb, ext3, google-soc, hurd, lvm, pkgsrc, rump, subfile, sysinst, syslog, teredo, ukfs, usb, uvc, wscons, wstablet]

The work's a very first cut, see Jared's email for more information.

While there, and if you're interested in this topic, also see Krister's email on his 's EuroBSDCon 2004 presentation (PDF version) entitled "Cross-compiling packages", which actually isn't so much about building packages at all, but really about running an emulator that can run foreign CPU core, and also heavily optimize it by detecting system calls and running them on the native kernel, plus even detecting some exec(2) calls, and running a native version of the called binary.

Jared also tells me that his work on NetBSD/usermode is closely related to Antti Kantee's Runnable Userspace Meta Programs (RUMPs) project, and I guess we'll see some more collaboration in that area coming soon...

[Tags: rump, usermode]

This year's project goes one step further as not only the VFS operations are passed from kernel to userspace, but the full filesystem code is ran in userspace, including memory management (buffer cache!). This allows developing filesystem code in userland with the "normal" kernel interfaces, but no panics if something goes wrong -- a simple core dump, and that's it. To cite from Antti's mail: ``If you want to give it a whirl, upgrade, go to sys/rump, type "make", go to fs/bin/$foofs and run the file system. It should work to a certain degree provided your system comes with puffs. Does it do anything a kernel file system wouldn't do? No .. except the core dump may be smaller and faster and less intrusive. Oh, actually it does one cool trick: you can run it on non-block devices, so you don't need to vnconfig the file system image first (provided, of course, that the file system starts right from the beginning of the image and that you wouldn't need a disklabel).''

For more information, see Antti's first and second email and his Summer-of-Code project page.

[Tags: google-soc, puffs, rump, ukfs]