His page describes the list of operating systems (and versions!) tested, a bit of configuration for each operating system, and the hardware and software versions used.

There are two charts available, the first one showing results for Ruby version 1.8.7, the second one with results for Ruby version 1.9. Please note that you can click on the graphs to get the corresponding ruby code - which is not commented too good, and thus leaves it to own interpretation on what exactly the test actually benchmarks.

Here's an example graph:

From a first look, NetBSD seems to be good for CPU-intensive applications (i.e. it has less overhead), but performance with virtual memory, regular expressions and threading can still be improved.

From looking at all the tests, it can be said that NetBSD makes a pretty good platform for Ruby deployment. For more, follow the discussion on the tech-perform list.

[Tags: benchmark, ruby]

Some details on the benchmarks:

• hackbench IRC server simulation:
  

• sysbench: MySQL OLTP simluation:
  

• build.sh: Compile benchmark:
  

SMP & audio: One area that hasn't been changed for moving towards fine-grained kernel locking was NetBSD's audio subsystem. As audio recording and playback is mostly done via interrupts, and as latency in those is critical, the audio subsystem was moved to the new interrupt handling system. The work can be found on the ad-audiomp branch, more information is available in Andrew's posting about the MP safe audio framework and drivers.

Benchmarking: Changing a system from inside out is a huge technical task. On the way, performance measurements and tuning are needed to make sure that the previous performance is still achieved while getting better performance in the desired development area. As a result, benchmarks results from Sun's libmicro benchmark suite were posted, which allow comparison not only against Linux and FreeBSD, but also between NetBSD-current and NetBSD 4.0, in order to identify if any bad effects were added. All performance tests were made on a machine with 8 CPUs, and the areas tested cover "small" (micro) areas like various system calls. Of course this doesn't lead to a 1:1 statement on how the systems will perform in a real-life scenario like e.g. in a database performance test, but it still help identifying problems and gives better hints where tuning can be done.

Another benchmark that was also made in that regard comes from Gregory McGarry, who has published performance measurements previously. This time, Gregory has run the lmbench 3.0 benchmark on recent NetBSD and FreeBSD systems as well as a number of previous NetBSD releases - useful for identifying performance degradation, too!

One other benchmark on dispatch latency run was made by Andrew Doran: on a machine that was (CPU-wise) loaded by some compile jobs, he started a two threads on a CPU that wasn't distracted by device interrupts, and measured how fast the scheduler reacted when one thread woke up the other one. The resulting graph shows that the scheduler handles the majority of requests in less than 10us - good enough for some realtime applications?

Kernel modules are another area that's under heavy change right now, and after recent changes to load modules from the bootloader and the kernel, the kernel build process was now changed so that pre-built kernel modules can be linked into a new kernel binary, resulting in a non-modular kernel. Eventually, this could mean that src/sys is built into separate modules, and that the (many) existing kernels that are present for each individual platform -- GENERIC, INSTALL is already gone, ALL, etc. etc. -- can be simply linked from pre-compiled modules, without recompiling things over again for each kernel. Of course the overal goal here is to speed up the system (and kernel!) build time, while maintaining maximum flexibility between modules and non-modular kernels.

With the progress in kernel modules, it is a question of time when the new kernel module handling supercedes the existing loadable kernel modules to such an extent that the latter will be completely removed from the system -- at least the latter was alredy proposed, but I'd prefer to see some documentation of the new system first. We'll see what comes first! (Documentation writers are always welcome! :-)

[Tags: benchmark, kernel, smp]

Andrew also updated his previous benchmarks, and posted about his recent results: ``Most of the sysbench runs that I've seen to date have sysbench running on the same machine as the database. That's a good test but with the exception of small installations and out-of-band activity, production setups rarely look like that. So I ran sysbench itself on a seperate dual core system.''

There are images that compare NetBSD 3 with NetBSD-current (where most of Andrew's changes are now), and NetBSD-current compared to Linux, FreeBSD, and OpenBSD:

The original benchmarks didn't include Solaris/x86, so Jaime Fournier sat down and repeated the test (on a single system). The results show that NetBSD beats Solaris by ~25% in the ReadOnly test, and that they're about on par in the ReadWrite test, with NetBSD kicking in earlier WRT the number of client threads, but Solaris keeping up longer before they both degrade. The courves are quite similar, and my guess is that there is some room for finetuning there:

Leaving aside all the arguments, Thor Lancelot Simon did some musing on the subject, and taking two identical MacMinis, he ran them with an application that's common to many of us: compiling stuff. "Stuff" in that case was building a release of the NetBSD 4.0 release branch, using NetBSD-current (as of may 2007), and FreeBSD 6.2RC2. The results?

• Build platform:   NetBSD 4.99.19 i386
  build.sh started: Tue May 15 18:24:20 EDT 2007
  build.sh ended:   Tue May 15 20:01:29 EDT 2007

• Build platform:   FreeBSD 6.2-RC2 i386
  build.sh started: Tue May 15 18:25:28 EDT 2007
  build.sh ended:   Tue May 15 20:20:47 EDT 2007
  

Maybe it should be noted that the building was done with 4 concurrent processes, to keep both cores of the MacMinis warm. On the NetBSD side, the SMP implementation is also the new "newlock2" based code by Andrew Doran that removes the kernel big lock for some paths inside the kernel, which may be responsible for the boost in performance.

See Thor's mail for more details.

(P.S.: This isn't intended as FreeBSD bashing, in case it's not obvious. Just a few facts...)

[Tags: benchmark, freebsd]

There are three results of interest here:

1. Amount of monitored websites by detected operating system:

   

   So out of ~13000 websites, it's only 238 (2%) that run NetBSD. Not terribly high, esp. when 774 sites run Minix (wtf?!), but well - it's lonely at the top. 8-)

2. Analyzing wesite performance statistcs by operating systems sorted by uptime:

   

   So over a period of at least three weeks, NetBSD showed best uptime results. Yai!

3. Analyzing websites performance statistics by operational systems sorted by response time:

   

   Again NetBSD scores the top position over three weeks when looking at the response times.

We hope this data will be valuable for our community in their hosting selection process.''

[Tags: benchmark, uptime]

• Linuxdays Chemnitz: I was only there for my presentation on sunday, due to not feeling too well the days before. Still, Stefan, Jörg, Charlie and many others staffed the booth just fine, and I think every single household in and around Chemnitz has a NetBSD install and/or Live CD now. :)

  Related talks to mention are Stefan Schumacher's talk on hardening systems with systrace and deleting data. My own talk was not too NetBSD specific, showing an application on how to implement dynamic DNS with some retail web/domain hoster. Slides for my talks are available as OpenOffice .ODP and as PDF. (I'll reconsider the move from TeX/prosper to OpenOffice after it was NOT as easy as I expected to find a machine running OOo for presentation purpose, after my laptop's harddisk crashed on the way to Chemnitz!)

• While at roadshows: Stefan Schumacher has made DIN A4 pkgsrc flyers in english and german language.

• NetBSD's puff-based FUSE implementation "refuse" is now in a state to also run the NTFS-g3 filesystem, which offers read/write support for NTFS. It's available from pkgsrc/filesystems/fuse-ntfs-3g.

• Google News found me an article that NetBSD stack supports Geode NAS design: ``Wasabi Systems Inc.'s BSD-based NAS (network attached storage) software stack now supports a Geode-based reference design from AMD. Wasabi Storage Builder for NAS, combined with AMD's Geode LX NAS RDK (reference design kit), provides a secure, reliable platform for the development of NAS devices, according to Wasabi. ''

  While that's all fine for Wasabi, it should be noted that whatever the company Wasabi offers is not automatically available in the freely available operating system called NetBSD. Integration efforts would have to happen first, so the headline of that article is unfortunately misleading if not to say plain wrong!

• Another article that's more to the point: Julio M. Merino Vidal has worked on getting multiboot support into NetBSD, and in his article ``Making NetBSD Multiboot-Compatible'' he talks more about it.

• Andrew Doran has done lots of work on NetBSD's thread and SMP implementation recently, and he has made a comparison between performance of the Scheduler-Activations-based code in NetBSD 4 and the one that will be in NetBSD 5 (AKA NetBSD-current, currently numbered as 4.99.13). See his mail to tech-kern or watch the images for 'make cleandir' on an empty source tree and the MySQL supersmack benchmark.

• Google runs another Summer of Code, and this year it's not clear upfront who will be allowed as mentoring organizations. NetBSD is ready to participate again, and there's an official announcement from NetBSD about this, including pointers to our suggested/wanted list of projects and the project application HowTo. People interested in submitting a project proposal (via google!) are encouraged to use the remaining time until the deadline to discuss their proposals on the public NetBSD tech-* lists! (Personally I'll try to stay out of GSoC this year to finish some reallife work. At least that's the plan so far ...)

• Three new security advisories were released:
  • NetBSD Security Advisory 2007-001: Integer overflow in ktruser()
  • NetBSD Security Advisory 2007-002: Integer overflows in Render and DBE extensions
  • NetBSD Security Advisory 2007-003: BIND multiple denial of service vulnerabilities

• Another article that doesn't mention NetBSD but g4u: ``How to Install a New Hard Drive: Tech Clinic'' by Joel Johnson. From the article: `` To make your new drive work like your old drive, you'll need a disk "cloner." There are a myriad of options, from commercial solutions such as the old favorite Ghost from Symantec ($70; symantec.com) and Copy Commander from VCom ($35; v-com.com) to free applications, such as MaxBlast from Maxtor, that come bundled with hard drives. If you're comfortable mucking around with Linux/BSD, I've had great luck with the free g4u application. If you have a local file server, you can even send the disk image from your laptop to an FTP site, install the larger drive, then FTP it back to your laptop, obviating the need for a drive enclosure''.

How much slower exactly using a disk image instead of a raw partition is was asked on tech-perform, and Greg Troxel gave some interesting numbers.

Now what I wonder is what impact the filesystem type has...

[Tags: benchmark, xen]

``I'm working on new benchmarks similar to my old ones, but I want to measure more this time. I have taken about 30GB of image data from a customer's webserver, and logged about 50.000 web requests, and wanted to replay them as fast as possibl. To measure the filesystem I want to keep 10% of the 30GB images seperately, and then use tar to unpack and measure. But the tests so far are very negative for BSD (10 minutes unpacking on Linux with ext3 against 26 minutes under FreeBSD 6 with UFS2. I'm therefore looking for benchmark ideas in this category (so not "install postgres and shove in the following selects"), which are friendly to BSD and unfriendly to Linux. There has to be something?! Please mail your hints to me!''

Given the last round of benchmarks and the improvements that were made to NetBSD after them, this sounds very interesting. Ideas? What is it that NetBSD can do better than Linux (and FreeBSD :-). Voice your opinion on the tech-perform mailing list!

[Tags: benchmark, fefe]