Insights on Release Engineering

This document was published in the January 2001 Daemonnews issue.

Insights on Open Source Release Engineering
or: How NetBSD 1.5 Was Born
Hubert Feyrer, December 2000

NetBSD 1.5 was released a few weeks ago, and after the dust has settled a bit now, we were able to get hold of Todd Vierling to tell us more about the 1.5 release engineering process. Todd was one of the release engineers who did most of the work in the "hot" phase of the 1.5 release cycle - here's an interview with him, and your chance to get some insight into the details of the release engineering process of a major Open Source project!

HF: How many people were there on the release engineering team, and what were their duties?

TV: The 1.5 branch release engineering team had nine members on it, and still prospectively retains eight of them for the 1.5.1 release, depending on who volunteers the time to do the job. That is, for the people who volunteer selflessly their available personal time.
Personally, I wasn't even expecting to be on the release engineering team. I was recruited for the task by Wasabi Systems (http://www.wasabisystems.com/), who devoted several weeks of my work time to the task of pushing NetBSD 1.5 out the door. That significantly helped wrapping up the release, and made me take notice that there is real industry interest, not just volunteer interest, in a product such as NetBSD.

HF: What were the major jobs during the release engineering cycle?

TV: The most time-consuming job was certainly pullups. CVS branch maintenance is not a pretty sight - particularly when the maintainer must also provide audit trails for large changes. We have scripts that partially automate this, but they certainly aren't perfect, and need input data from the pullup requests in some very specific formats. This is a place where better tools will definitely find a place in NetBSD developer use in the near future.
I also made sure that, while I was executing pullup requests, I personally read diffs of the merged code (and compiled it, where feasible to my available host systems) before committing to the branch. I don't necessarily understand all the code innards intimately, but extra eyeballs and compilers help to catch the occasional mechanical merge error.
Besides pullups, there was the coordination of the build team for all the architectures, but I'll get to that in a moment.
Other tasks included arranging the files on the FTP server using a new layout that helps us reduce distribution space for architectures that share the same base CPU and executable format; preparing the release announcement (which Charles Hannum revised for final release); and preparing the first officially supported, downloadable NetBSD CD-ROM images, which I'll also mention later.

HF: The release was delayed a bit, what were the reasons for this?

TV: The concept of "delayed" in proportion to the 1.5 release is probably a bit skewed, and I'm partly to blame for saying it was delayed for a "long time". NetBSD 1.5 has so many new features that even I can't recount more than one-sixth of them without resorting to the release announcement and change summary, and it's still reasonably close to its target release (end of September) compared to some prior NetBSD releases.
NetBSD's biggest release impediment is also its most important feature: we released 1.5 on 10 base CPU types comprising 20 groups of hardware architectures ("ports"). Taking in to account I/O buses, MMUs, and system controllers that another OS would consider to be different platforms, the actual number of specific hardware platforms is somewhere closer to 40. That's no small feat, and next to CP/M (which used the same CPU type throughout), NetBSD runs on more hardware platforms than any other full-featured OS in history.
Part of the slowdown was that most of the people working on release engineering were doing the work on personal time, making the available time for the task unpredictable. This will probably always be a small problem, but we have some amazingly devoted volunteers, and now some background business support as well.
Toward the end of the release cycle, we decided to delay a few days here and there to pull in and test some very important security fixes to the IPv6 network tunnel interfaces, BIND (which was updated before release to 8.2.2-P7), and tools throughout the tree in a compiler-aided printf() formatting string sweep. All though the release cycle, we did bring in many minor yet important functionality improvements for dozens of tools from the development sources into the 1.5 release.
As an unexpected benefit from the longer release cycle, though, NetBSD 1.5 has had a longer testing cycle than prior major releases.

HF: So how did you know when it was the "right" time to do the release at the time it happened, and not delayed for another feature?

TV: The NetBSD developer base is quite diverse, and opinions about the "right" time to release have been just as different from one developer to another. I had my own personal crisis when I was recruited to help out - I'm a bit of a perfectionist when developing my own software, and I was placed in the contrasting position of keeping the NetBSD 1.5 release cycle as close to schedule as feasible.
We went through several "beta release" cycles, each one shorter than the prior one, and released after a beta cycle that involved nearly no machine independent pullups. We could have gone through more beta cycles ad infinitum, but the time came to say, officially, that the system was up to the task of production use, and any further changes would wait until 1.5.1. We're still finding and fixing more bugs all the time, of course.
This is a learning exercise, and we're working on improving the process. I have a feeling the release cycle will keep more to a schedule in 1.6, and future patch releases such as 1.5.1 will take quite a bit less time to accomplish.

HF: With the many different machines supported by NetBSD, how was the binary distribution available on the FTP server created? Did you have all machines available and do all the builds yourself?

TV: Though NetBSD can cross-build on a number of platforms now, we don't have a solid procedure yet to use for building releases with a cross compiler. So, even on VAXen, we currently build natively on almost every platform we support. (The VAX example is a bit tongue-in-cheek; our VAX builder, Matt Thomas, built the whole system and uploaded it within a day. Most m680x0 platforms took nearly three times as long.)
16 people did the actual building, with two handling more than one port's build. Though some port maintainers were too occupied to do the builds personally, I found more than enough volunteers to take up those jobs. The actual build time was only a few days from tag to completion, and all platforms were building simultaneously.
NetBSD 1.5 also increased in binary size a bit from 1.4 and prior releases. Feeping creaturism, or something like that. The typical base binary release - gzip compressed - is right around 70MB including X11 binaries, and that's not counting the third-party packages from NetBSD "pkgsrc".

HF: Some ISO images were put up recently on the FTP site, after the release was available for some time. You said you have some special ideas to create them on the fly?

TV: Before I became directly involved in NetBSD 1.5 release engineering, Wasabi Systems also wanted to put together a CD-ROM distribution. Also, the demand for CD-ROM images from the FTP-downloading public has been extremely high lately, thanks to ever-increasing available bandwidth for client users.
In hopes that I could address both these issues, I put together a flexible CD-ROM image building tool that could take any combination of architectures and put them together with a minimum of duplicated code and a maximum of compatibility with host OS's such as Windows, MacOS, and AmigaOS. It also makes the CDs bootable on up to 7 architectures (thus far). NetBSD source users can find it in NetBSD-current under src/distrib/cdrom, and this tool will be more fully documented on the NetBSD website later this week.
There's more to the issue of downloadable ISOs, though. CD-ROM images, if downloadable, contain a huge amount of redundancy. Over 98% of NetBSD CD-ROM data is simply duplication of the gzipped distribution files available elsewhere on the FTP site - a huge waste of diskspace, especially if mirror sites want to pick up the images, and if we were to provide images for multiple releases or multiple combinations of platforms.
I have been working on a side project to find a way to reduce this redundancy. The result is a new tool, to be released at the end of this week at http://www.duh.org/expack/, which strips the redundant file data from the file content, reducing image size to no more than a few MB (and as little as 400KB on non-bootable architectures). At download time, ftpd runs a special filter program that interpolates the redundant data into a fully populated image, with checksum verification. This filter has little more system overhead than "cat". With this tool, FTP servers can provide CD-ROM images for download, while using up only 40MB of extra diskspace for all architectures.

HF: The ISO images on ftp.netbsd.org contain one architecture per CD. Will there be a CD (or CD-set) with all architectures?

TV: Multiple-architecture CD-ROM images may be available unofficially on the ftp.netbsd.org website as a trial run of the expack tool I mentioned before. However, the easiest method for obtaining NetBSD on CD-ROM, particularly for multiple architectures, is to buy a prepressed distribution.
A list of all known CD vendors is available at http://www.netbsd.org/Sites/cdroms.html. One thing to note about NetBSD distributions is that - unlike Linux - all NetBSD distributions use the same source code tree. Some CD vendors build the sources themselves, but most use the same binaries that the NetBSD Foundation releases on ftp.netbsd.org.

HF: If you choose to do release engineering for the next release again, what will you do differently?

TV: Well, I am staying on the 1.5 release engineering team to help put out a 1.5.1 patch release, hopefully sometime in February. The patch releases aren't aiming for the kind of functionality differences between NetBSD 1.4 and 1.5; they mainly just fix problems found after the release or fixed in the interim.
I noted before that NetBSD 1.5 runs on 10 CPUs on 20 ports. The NetBSD-current development sources are fast approaching 14 base CPU types and 40 ports. We're expecting about 30 of those ports to be part of the next major NetBSD release (tentatively numbered 1.6). That release won't be a simple one.
With that said, I may stick around to volunteer for 1.6 release engineering. Tedium aside, this has been a rather fun thing to do.

HF: Do you have an idea what the NetBSD users can look forward to for NetBSD 1.6?

TV: Probably the most noticeable feature will be some support for symmetrical multiprocessing on i386 (and possibly other) platforms. NetBSD has been a little behind in SMP development from the other BSD's, but I have to mention again that because of the number of platforms NetBSD supports, we try Very Hard to come up with abstractions that work on more than just one platform. SMP may not be entirely complete in 1.6, but there's quite a bit in the NetBSD-current development tree as I write this; it looks very promising.
There's also active work on making the NetBSD base system more granular, using our existing packaging infrastructure used for "pkgsrc". It's likely that NetBSD 1.6 will be installable using a set of packages much like commercial Unix-like OS's, allowing admins to install as much or as little of the base system as desired.
Some platforms will be making the switch from the historic a.out binary format to the more well supported ELF format. And another project ("UBC") has been integrated that unifies the concepts of page cache and buffer cache, both increasing general system speed and correctly fixing the historic problem of consistency between mmap() and regular file access.
Of course, there will be more platforms as I mentioned earlier. They include, among others, Hitachi Super-H system-evaluation boards and the Sega Dreamcast, SGI workstations, more UltraSPARC systems, and a few older systems including the OMRON LUNA m68k's, original MIPS Corporation workstations, and the Acorn Archimedes. One developer also plans to bring back the BeBox port to an official release.

HF: Todd, thanks a lot for your time. Is there any final word that you'd like to tell to the DaemonNews readers and/or the NetBSD community out there?

TV: Sure. If you haven't tried NetBSD yet, you don't know what you're missing. http://www.netbsd.org/

HF:	How many people were there on the release engineering team, and what were their duties?
TV:	The 1.5 branch release engineering team had nine members on it, and still prospectively retains eight of them for the 1.5.1 release, depending on who volunteers the time to do the job. That is, for the people who volunteer selflessly their available personal time. Personally, I wasn't even expecting to be on the release engineering team. I was recruited for the task by Wasabi Systems (http://www.wasabisystems.com/), who devoted several weeks of my work time to the task of pushing NetBSD 1.5 out the door. That significantly helped wrapping up the release, and made me take notice that there is real industry interest, not just volunteer interest, in a product such as NetBSD.
HF:	What were the major jobs during the release engineering cycle?
TV:	The most time-consuming job was certainly pullups. CVS branch maintenance is not a pretty sight - particularly when the maintainer must also provide audit trails for large changes. We have scripts that partially automate this, but they certainly aren't perfect, and need input data from the pullup requests in some very specific formats. This is a place where better tools will definitely find a place in NetBSD developer use in the near future. I also made sure that, while I was executing pullup requests, I personally read diffs of the merged code (and compiled it, where feasible to my available host systems) before committing to the branch. I don't necessarily understand all the code innards intimately, but extra eyeballs and compilers help to catch the occasional mechanical merge error. Besides pullups, there was the coordination of the build team for all the architectures, but I'll get to that in a moment. Other tasks included arranging the files on the FTP server using a new layout that helps us reduce distribution space for architectures that share the same base CPU and executable format; preparing the release announcement (which Charles Hannum revised for final release); and preparing the first officially supported, downloadable NetBSD CD-ROM images, which I'll also mention later.
HF:	The release was delayed a bit, what were the reasons for this?
TV:	The concept of "delayed" in proportion to the 1.5 release is probably a bit skewed, and I'm partly to blame for saying it was delayed for a "long time". NetBSD 1.5 has so many new features that even I can't recount more than one-sixth of them without resorting to the release announcement and change summary, and it's still reasonably close to its target release (end of September) compared to some prior NetBSD releases. NetBSD's biggest release impediment is also its most important feature: we released 1.5 on 10 base CPU types comprising 20 groups of hardware architectures ("ports"). Taking in to account I/O buses, MMUs, and system controllers that another OS would consider to be different platforms, the actual number of specific hardware platforms is somewhere closer to 40. That's no small feat, and next to CP/M (which used the same CPU type throughout), NetBSD runs on more hardware platforms than any other full-featured OS in history. Part of the slowdown was that most of the people working on release engineering were doing the work on personal time, making the available time for the task unpredictable. This will probably always be a small problem, but we have some amazingly devoted volunteers, and now some background business support as well. Toward the end of the release cycle, we decided to delay a few days here and there to pull in and test some very important security fixes to the IPv6 network tunnel interfaces, BIND (which was updated before release to 8.2.2-P7), and tools throughout the tree in a compiler-aided printf() formatting string sweep. All though the release cycle, we did bring in many minor yet important functionality improvements for dozens of tools from the development sources into the 1.5 release. As an unexpected benefit from the longer release cycle, though, NetBSD 1.5 has had a longer testing cycle than prior major releases.
HF:	So how did you know when it was the "right" time to do the release at the time it happened, and not delayed for another feature?
TV:	The NetBSD developer base is quite diverse, and opinions about the "right" time to release have been just as different from one developer to another. I had my own personal crisis when I was recruited to help out - I'm a bit of a perfectionist when developing my own software, and I was placed in the contrasting position of keeping the NetBSD 1.5 release cycle as close to schedule as feasible. We went through several "beta release" cycles, each one shorter than the prior one, and released after a beta cycle that involved nearly no machine independent pullups. We could have gone through more beta cycles ad infinitum, but the time came to say, officially, that the system was up to the task of production use, and any further changes would wait until 1.5.1. We're still finding and fixing more bugs all the time, of course. This is a learning exercise, and we're working on improving the process. I have a feeling the release cycle will keep more to a schedule in 1.6, and future patch releases such as 1.5.1 will take quite a bit less time to accomplish.
HF:	With the many different machines supported by NetBSD, how was the binary distribution available on the FTP server created? Did you have all machines available and do all the builds yourself?
TV:	Though NetBSD can cross-build on a number of platforms now, we don't have a solid procedure yet to use for building releases with a cross compiler. So, even on VAXen, we currently build natively on almost every platform we support. (The VAX example is a bit tongue-in-cheek; our VAX builder, Matt Thomas, built the whole system and uploaded it within a day. Most m680x0 platforms took nearly three times as long.) 16 people did the actual building, with two handling more than one port's build. Though some port maintainers were too occupied to do the builds personally, I found more than enough volunteers to take up those jobs. The actual build time was only a few days from tag to completion, and all platforms were building simultaneously. NetBSD 1.5 also increased in binary size a bit from 1.4 and prior releases. Feeping creaturism, or something like that. The typical base binary release - gzip compressed - is right around 70MB including X11 binaries, and that's not counting the third-party packages from NetBSD "pkgsrc".
HF:	Some ISO images were put up recently on the FTP site, after the release was available for some time. You said you have some special ideas to create them on the fly?
TV:	Before I became directly involved in NetBSD 1.5 release engineering, Wasabi Systems also wanted to put together a CD-ROM distribution. Also, the demand for CD-ROM images from the FTP-downloading public has been extremely high lately, thanks to ever-increasing available bandwidth for client users. In hopes that I could address both these issues, I put together a flexible CD-ROM image building tool that could take any combination of architectures and put them together with a minimum of duplicated code and a maximum of compatibility with host OS's such as Windows, MacOS, and AmigaOS. It also makes the CDs bootable on up to 7 architectures (thus far). NetBSD source users can find it in NetBSD-current under src/distrib/cdrom, and this tool will be more fully documented on the NetBSD website later this week. There's more to the issue of downloadable ISOs, though. CD-ROM images, if downloadable, contain a huge amount of redundancy. Over 98% of NetBSD CD-ROM data is simply duplication of the gzipped distribution files available elsewhere on the FTP site - a huge waste of diskspace, especially if mirror sites want to pick up the images, and if we were to provide images for multiple releases or multiple combinations of platforms. I have been working on a side project to find a way to reduce this redundancy. The result is a new tool, to be released at the end of this week at http://www.duh.org/expack/, which strips the redundant file data from the file content, reducing image size to no more than a few MB (and as little as 400KB on non-bootable architectures). At download time, ftpd runs a special filter program that interpolates the redundant data into a fully populated image, with checksum verification. This filter has little more system overhead than "cat". With this tool, FTP servers can provide CD-ROM images for download, while using up only 40MB of extra diskspace for all architectures.
HF:	The ISO images on ftp.netbsd.org contain one architecture per CD. Will there be a CD (or CD-set) with all architectures?
TV:	Multiple-architecture CD-ROM images may be available unofficially on the ftp.netbsd.org website as a trial run of the expack tool I mentioned before. However, the easiest method for obtaining NetBSD on CD-ROM, particularly for multiple architectures, is to buy a prepressed distribution. A list of all known CD vendors is available at http://www.netbsd.org/Sites/cdroms.html. One thing to note about NetBSD distributions is that - unlike Linux - all NetBSD distributions use the same source code tree. Some CD vendors build the sources themselves, but most use the same binaries that the NetBSD Foundation releases on ftp.netbsd.org.
HF:	If you choose to do release engineering for the next release again, what will you do differently?
TV:	Well, I am staying on the 1.5 release engineering team to help put out a 1.5.1 patch release, hopefully sometime in February. The patch releases aren't aiming for the kind of functionality differences between NetBSD 1.4 and 1.5; they mainly just fix problems found after the release or fixed in the interim. I noted before that NetBSD 1.5 runs on 10 CPUs on 20 ports. The NetBSD-current development sources are fast approaching 14 base CPU types and 40 ports. We're expecting about 30 of those ports to be part of the next major NetBSD release (tentatively numbered 1.6). That release won't be a simple one. With that said, I may stick around to volunteer for 1.6 release engineering. Tedium aside, this has been a rather fun thing to do.
HF:	Do you have an idea what the NetBSD users can look forward to for NetBSD 1.6?
TV:	Probably the most noticeable feature will be some support for symmetrical multiprocessing on i386 (and possibly other) platforms. NetBSD has been a little behind in SMP development from the other BSD's, but I have to mention again that because of the number of platforms NetBSD supports, we try Very Hard to come up with abstractions that work on more than just one platform. SMP may not be entirely complete in 1.6, but there's quite a bit in the NetBSD-current development tree as I write this; it looks very promising. There's also active work on making the NetBSD base system more granular, using our existing packaging infrastructure used for "pkgsrc". It's likely that NetBSD 1.6 will be installable using a set of packages much like commercial Unix-like OS's, allowing admins to install as much or as little of the base system as desired. Some platforms will be making the switch from the historic a.out binary format to the more well supported ELF format. And another project ("UBC") has been integrated that unifies the concepts of page cache and buffer cache, both increasing general system speed and correctly fixing the historic problem of consistency between mmap() and regular file access. Of course, there will be more platforms as I mentioned earlier. They include, among others, Hitachi Super-H system-evaluation boards and the Sega Dreamcast, SGI workstations, more UltraSPARC systems, and a few older systems including the OMRON LUNA m68k's, original MIPS Corporation workstations, and the Acorn Archimedes. One developer also plans to bring back the BeBox port to an official release.
HF:	Todd, thanks a lot for your time. Is there any final word that you'd like to tell to the DaemonNews readers and/or the NetBSD community out there?
TV:	Sure. If you haven't tried NetBSD yet, you don't know what you're missing. http://www.netbsd.org/