Interview with Jun-ichiro Hagino about the KAME Project

This document was published on O'Reilly's ONlamp.com. See the Slashdot announce!

Interview with Jun-ichiro "Itojun" Hagino
about the KAME Project
Hubert Feyrer, January 2001

The KAME IPv6 stack is very well known in the BSD world and beyond, but noone hardly knows what's behind the KAME project and what the people behind it are. We grabbed one of the core developers / integrators of the KAME project, Jun-ichiro "itojun" Hagino, and asked him to tell us all about it.

HF: When was the idea of the KAME project born, what's it's history?

I: In Japan, we have a internet technology reserch group called "WIDE", which has a large number of researchers from universities and companies. We operate our research network backbone, and do random internet researches. Think of it as Japanese version of (USENIX + Internet2) / 2.
In 1995, we as WIDE started IPv6 implementation and test operations. We thought IPv6 is cool, and we also thought we have many things to be solved. IPng effort was first started at INET92 conference held in Kobe, Japan, so it is natural for Japanese guys to get involved :-). Many people got interested, and in WIDE group we had more than 5 implementations. Some of these implementations are from companies, and some are from universities.
In 1997, we became aware of two problems. One was the maintenance of these IPv6 stacks. These implementations had only one or two developers for each, so we were wasting our human resources maintaining too many IPv6 stacks. We thought it is better to integrate implementations, taking best parts from each of them, and ship a single source code tree. Another problem is availability of the code to wider audiences. We learnt IPv4 and its gory details mainly from BSD source code, not from specifications. We thought it is very important to integrate well-documented IPv6 source code into BSD, for teaching IPv6 to people, deploying IPv6, and torture-testing IPv6 and making feedback to the specification. We felt the serious need for deploying IPv6 code into BSDs.
We formed KAME project in April 1998 to solve the above two problems. KAME is a consortium of companies and universities. KAME implementers are those guys who implemented IPv6 stack in their companies and universities. The year 1998 was basically spent for integration of Japanese codebases. After that, we are doing BSD integrations and new developments. Now we support 7 different versions of BSDs (freebsd[234], bsdi[34], netbsd, openbsd), and our code is integrated into all of the latest BSD releases.
http://www.kame.net/project-overview.html has slightly more "official" answer :-).

HF: How many people are there on the Kame project, and what are they doing?

I: We have about 10 implementers at KAME. Though the base specifications were finished, IPv6 is still in its infancy. We see many new APIs and features coming, and we are improving many things. So we still are developing new items and fixing bugs. Examples are like scoped routing, better multihoming, DNS enhancements, PPP (address assignment issues are very hard), routing (scalability issues), multicast, and many other things.
Another major portion of work is to backport changes to BSDs. This task is very difficult IMHO. Because some of the APIs/protocols/code are not mature enough for general consumption, we carefully need to decide which portion to get backported.
We also have many supporting casts, including managements (bosses in participating companies and university faculties) and user community like mailing list. Without their support and cooperation KAME cannot exist so I would like to thank them now.

HF: Who pays for your travel, office space, machines & connectivity etc.?

I: Again KAME is a consortium of companies and universities. Developers come from these companies. Participating companies can get expertise from KAME earlier than others, and can understand more details. They can also use IPv6-ready BSD operating systems in their routers, which can reduce labors in these companies. Also the BSD IPv6 stack is guaranteed to be specification conformant, because of tests we go through.
For example, I'm with IIJ, one of the largest ISP in Japan. IIJ is the most aggressive ISP in deploying IPv6, among the ISPs in the world. My paycheck and travel tickets come from IIJ, and IIJ gets expertise and KAME-ready BSDs. IIJ makes and sells routers, IIJ also uses lots of BSD boxes for servers.
KAME has its own office, machines and connectivity. In addition to support from companies, we get research funding for our work. Basically these facilities are maintained by budget from companies, and from research fundings.

HF: you tell us something about interoperability test you do?

I: KAME has a sister project called TAHI. TAHI is focused on IPv6 and IPsec conformance and interoperability tests, and torture-test our code. If we put some bugs in KAME tree, they will notice that and inform us. The test tool is freely available at http://www.tahi.org/, so you can use it in your office/home.
TAHI activity is helping many people, including commercial vendors and free software developers. For IPv6 to be successful we need a good conformance/interoperability test suites. For example, there are people trying to improve Linux IPv6 support, based on TAHI test results (http://www.linux-ipv6.org/).
We also have attended various interoperability test events, including UNH IOL (http://www.iol.unh.edu/), connectathon (http://www.connectathon.org/), and a bunch of IPsec bakeoffs.

HF: Besides the IPv6 stack, there's also an AltQ implementation. What other (sub)projects does the KAME project have in the pipe?

I: At KAME and WIDE, we do other cool stuffs other than IPv6 too. For example, we do IPsec and routing enhancements at KAME. Other people at WIDE (outside of KAME) are also doing cool things like diffserv, MPLS and mobile-ip6. Additional role of KAME is to become the clearing house for these developments, and provide these technologies to all of the BSDs.
Since KAME needs to support 7 different version/variants of BSDs, we have good expertise in sharing networking code among these BSDs. It is not that easy as these BSDs are highly diverged at this point. ALTQ has been separately maintained before, but now it is integrated into KAME tree. It was because Kenjiro Cho, the author of ALTQ, find it much easier to work on ALTQ in KAME tree, to support multiple BSDs.

HF: Do you (Kame) have any plans to work on NFSv4?

I: We do not do much about NFSv4. I believe Dug Song (of OpenBSD/dsniff) is working on it so I hope to see it available for all BSDs.

HF: Let's talk a bit more about history. In the early days of IPv6, there were other IPv6 implementations that were directed towards BSD - INRIA and NRL come to mind. Aparently they disappeared - was the code merged into KAME, are any of the developers of these projects participate on the KAME project now?

I: In year 1998 and 1999, we tried to integrate NRL, INRIA nad KAME code into one. It was called the "unified-ipv6" project. It was a very difficult effort as we designed everything slightly different from each other! But it was a good effort, we exchanged bugfixes and some portion of code among us.
At some point during the unified-ipv6 effort, NRL and INRIA projects disbanded due to research funding issues. NRL and INRIA codebases are not actively maintained/distributed any more (Francis Dupont, who was in INRIA IPv6 project, still maintains son-of-INRIA implementation). It would be correct to say that KAME is now "unified-ipv6" by itself, as KAME tree has some NRL and INRIA code in it.
NRL/INRIA guys are not part of KAME team at this point. They are highly active in BSD and IPv6 mailing lists, we exchange ideas very much.

HF: How did you get to join the Kame project, what did you do before that?

I: I was a doctor candidate at Keio university, as well as a researcher at Sony laboratory. I did object-oriented operating system researches together with other researchers at Sony. Sony dog robot ("AIBO") uses a descendant of the opreating system.
I joined KAME and IIJ in April 1998 (IIJ is a participant of KAME). At IIJ I help them deploy IPv6 network, and provide them with IPv6 and other advanced networking technologies based on KAME work.

HF: There's the rumour that you never sleep and the only time that you don't read mail is when you're answering someone else's mail. Is that true? ;-)

I: I got a couple of doppelgangers :-). Seriously, I do sleep every day! If you carefully check email timestamps, I almost never send emails between 3AM to 8AM, Japan time.

HF: Do you work on other projects than KAME/IPv6? What's your favourite pastime, hobby, ...?

I: My favorite area of hacking other than OS/network are like multilingualization and mobile devices. As a native Japanese speaker it is critical for me to have a good multilingualized software, so I work on it :-). I'm a developer of multilingual nvi, as well as citrus BSD locale framework. I used to collect digital cameras and shoot random pictures. These days I do not play with digital cameras, maybe I should do this again.
Pastime... I love computers too much, it is a problem. I feel so comfortable when I'm typing :-). Other than computers, I like foods, both hot foods (like Indian, Thai) and non-hot foods. I love movies very much, DVDs are piling up in my room. Recently I saw lots of Korean movies. I highly recommend "Christmas in August". I'm an indoor guy:-)

HF: Besides your work on bringing IPv6 into various BSDs, you are working on the OpenBSD project and are also a core member of the NetBSD projects. What's your jobs there, do you participate in these projects beyond merging the IPv6 code into the existing network stacks?

I: I'm a developer for NetBSD, OpenBSD and FreeBSD. I don't think there's too many people with commit accesses to 3 BSDs, I believe more people should do this :-). Part of my task in KAME is to backporting fixes made by KAME into NetBSD and OpenBSD. So I do a lot of IPv6 integration work in OpenBSD.
In the OpenBSD community, beyond IPv6 integration, I work on security issues in networking code, and sometimes other portions. There are lots of work to be done - it is amazing we still find bugs in BSD IPv4 code, even though it was implemented so long ago. I get so much help from OpenBSD folks, in term of security perspective.
I always try to backport changes I made to a BSD, to other *BSDs. So, security changes I made on OpenBSD will get ported back to KAME tree, and other BSDs (I'm very sure that it will get ported back to NetBSD). I find good thing and bad thing in having multiple BSD codebases. Good thing is that we have projects with different goals, stimulating and encouraging each other. One negative thing are that manpower is splitted, code becomes diverged and becomes harder to port. I think I'm trying to solve part of the negative side.

HF: What about FreeBSD, BSDi and Darwin, is there someone working on these projects like you work on NetBSD?

I: FreeBSD and BSD/OS also integrate KAME. FreeBSD guys like Kris Kennaway and Hajime Umemoto helps us synchronize FreeBSD IPv6 code and KAME tree. For BSD/OS, since we cannot look at the source code, a BSDi guy is doing the integration, and we comment on the integration looking at beta releases.
It seems that there are people working on KAME integration into Darwin/MacOS X. I really hope to see MacOS X shipped with IPv6 support, that will boost IPv6 userbase.

HF: General acceptance of of IPv6 seems to be very low as people don't feel a pressing need to migrate from IPv4. When do you think will 25%, 50%, 100% of the internet sites talk IPv6?

I: I'm not very sure, but if you are talking about "IPv6 enabled devices" UNIX boxes are becoming more and more IPv6 ready. If you install Solaris 8, it is IPv6 ready. I heard that next official release of Windows2K (2001?) will be IPv6 ready. People just need to enable it, or configure it, that's all. So I see obstacle in ISP side, and router vendors side. They have kind of deadlock situation: ISPs do not deploy IPv6 until routers get ready, and vendors do not ship routers until ISPs deploy it. I would like to change the situation by making code more available.
I don't understand why people are keep using NAT, for me they are wasting their time! I had more than enough troubles with NAT, I really need simpler solution - that is, IPv6. Internet technology has to be much more stable, simple, and robust, unlike configurations with NAT. I believe people need to get informed better.

HF: Where do you see applications of IPv6 that can speed up it's deployment, what's the "killer application"?

I: To deploy IPv6, I believe we just need to make sure more products are IPv6 ready, and more number of ISPs are IPv6 ready (or at least their equipments are). This is what we do now.
Some says that IPv4 was deployed because of WWW, and call WWW "the killer application". I feel the story is too simplified. IPv4 has been there even before WWW hit the street. There were lots of ISPs, there were university networks, IPv4 was deployed enough to hit the critical mass. Then, WWW appeared and IPv4 deployed to much more wider people.
So, what I believe is we need to deploy IPv6 enough first, and then someone can come up with an application that is not possible with IPv4. I do not really have concrete image about how that application will look like. IPv6 can be used for much wider purposes, due to bigger addresses, including cellphones, lightbulbs, vehicles and household appliances, it will make a big difference.
From implementation and specification point of view, IPv6 is ready to replace IPv4. I do everything I would do with IPv4, by IPv6. I don't have to deal with NAT, I don't have to worry about address space shortage any more, it's just better. We just need wider availability for it.

HF: Is there anything left you'd like to tell our readers?

I: If you have latest releases of NetBSD, OpenBSD or FreeBSD, you have IPv6 builtin, enabled by default. Try "telnet localhost" and you are using IPv6! The best way to learn about IPv6 is to actually use it, so please try it out, and do not hesitate to ask questions at users@ipv6.org.
IMHO Japan is the most cool place for BSD geeks, and geeks in general. There are magazines dedicated to BSD, shops specialized in BSDs and UNIX, and coolest laptops and PDAs. Please visit Japan when you have a chance.

HF:	When was the idea of the KAME project born, what's it's history?
I:	In Japan, we have a internet technology reserch group called "WIDE", which has a large number of researchers from universities and companies. We operate our research network backbone, and do random internet researches. Think of it as Japanese version of (USENIX + Internet2) / 2. In 1995, we as WIDE started IPv6 implementation and test operations. We thought IPv6 is cool, and we also thought we have many things to be solved. IPng effort was first started at INET92 conference held in Kobe, Japan, so it is natural for Japanese guys to get involved :-). Many people got interested, and in WIDE group we had more than 5 implementations. Some of these implementations are from companies, and some are from universities. In 1997, we became aware of two problems. One was the maintenance of these IPv6 stacks. These implementations had only one or two developers for each, so we were wasting our human resources maintaining too many IPv6 stacks. We thought it is better to integrate implementations, taking best parts from each of them, and ship a single source code tree. Another problem is availability of the code to wider audiences. We learnt IPv4 and its gory details mainly from BSD source code, not from specifications. We thought it is very important to integrate well-documented IPv6 source code into BSD, for teaching IPv6 to people, deploying IPv6, and torture-testing IPv6 and making feedback to the specification. We felt the serious need for deploying IPv6 code into BSDs. We formed KAME project in April 1998 to solve the above two problems. KAME is a consortium of companies and universities. KAME implementers are those guys who implemented IPv6 stack in their companies and universities. The year 1998 was basically spent for integration of Japanese codebases. After that, we are doing BSD integrations and new developments. Now we support 7 different versions of BSDs (freebsd[234], bsdi[34], netbsd, openbsd), and our code is integrated into all of the latest BSD releases. http://www.kame.net/project-overview.html has slightly more "official" answer :-).
HF:	How many people are there on the Kame project, and what are they doing?
I:	We have about 10 implementers at KAME. Though the base specifications were finished, IPv6 is still in its infancy. We see many new APIs and features coming, and we are improving many things. So we still are developing new items and fixing bugs. Examples are like scoped routing, better multihoming, DNS enhancements, PPP (address assignment issues are very hard), routing (scalability issues), multicast, and many other things. Another major portion of work is to backport changes to BSDs. This task is very difficult IMHO. Because some of the APIs/protocols/code are not mature enough for general consumption, we carefully need to decide which portion to get backported. We also have many supporting casts, including managements (bosses in participating companies and university faculties) and user community like mailing list. Without their support and cooperation KAME cannot exist so I would like to thank them now.
HF:	Who pays for your travel, office space, machines & connectivity etc.?
I:	Again KAME is a consortium of companies and universities. Developers come from these companies. Participating companies can get expertise from KAME earlier than others, and can understand more details. They can also use IPv6-ready BSD operating systems in their routers, which can reduce labors in these companies. Also the BSD IPv6 stack is guaranteed to be specification conformant, because of tests we go through. For example, I'm with IIJ, one of the largest ISP in Japan. IIJ is the most aggressive ISP in deploying IPv6, among the ISPs in the world. My paycheck and travel tickets come from IIJ, and IIJ gets expertise and KAME-ready BSDs. IIJ makes and sells routers, IIJ also uses lots of BSD boxes for servers. KAME has its own office, machines and connectivity. In addition to support from companies, we get research funding for our work. Basically these facilities are maintained by budget from companies, and from research fundings.
HF:	you tell us something about interoperability test you do?
I:	KAME has a sister project called TAHI. TAHI is focused on IPv6 and IPsec conformance and interoperability tests, and torture-test our code. If we put some bugs in KAME tree, they will notice that and inform us. The test tool is freely available at http://www.tahi.org/, so you can use it in your office/home. TAHI activity is helping many people, including commercial vendors and free software developers. For IPv6 to be successful we need a good conformance/interoperability test suites. For example, there are people trying to improve Linux IPv6 support, based on TAHI test results (http://www.linux-ipv6.org/). We also have attended various interoperability test events, including UNH IOL (http://www.iol.unh.edu/), connectathon (http://www.connectathon.org/), and a bunch of IPsec bakeoffs.
HF:	Besides the IPv6 stack, there's also an AltQ implementation. What other (sub)projects does the KAME project have in the pipe?
I:	At KAME and WIDE, we do other cool stuffs other than IPv6 too. For example, we do IPsec and routing enhancements at KAME. Other people at WIDE (outside of KAME) are also doing cool things like diffserv, MPLS and mobile-ip6. Additional role of KAME is to become the clearing house for these developments, and provide these technologies to all of the BSDs. Since KAME needs to support 7 different version/variants of BSDs, we have good expertise in sharing networking code among these BSDs. It is not that easy as these BSDs are highly diverged at this point. ALTQ has been separately maintained before, but now it is integrated into KAME tree. It was because Kenjiro Cho, the author of ALTQ, find it much easier to work on ALTQ in KAME tree, to support multiple BSDs.
HF:	Do you (Kame) have any plans to work on NFSv4?
I:	We do not do much about NFSv4. I believe Dug Song (of OpenBSD/dsniff) is working on it so I hope to see it available for all BSDs.
HF:	Let's talk a bit more about history. In the early days of IPv6, there were other IPv6 implementations that were directed towards BSD - INRIA and NRL come to mind. Aparently they disappeared - was the code merged into KAME, are any of the developers of these projects participate on the KAME project now?
I:	In year 1998 and 1999, we tried to integrate NRL, INRIA nad KAME code into one. It was called the "unified-ipv6" project. It was a very difficult effort as we designed everything slightly different from each other! But it was a good effort, we exchanged bugfixes and some portion of code among us. At some point during the unified-ipv6 effort, NRL and INRIA projects disbanded due to research funding issues. NRL and INRIA codebases are not actively maintained/distributed any more (Francis Dupont, who was in INRIA IPv6 project, still maintains son-of-INRIA implementation). It would be correct to say that KAME is now "unified-ipv6" by itself, as KAME tree has some NRL and INRIA code in it. NRL/INRIA guys are not part of KAME team at this point. They are highly active in BSD and IPv6 mailing lists, we exchange ideas very much.
HF:	How did you get to join the Kame project, what did you do before that?
I:	I was a doctor candidate at Keio university, as well as a researcher at Sony laboratory. I did object-oriented operating system researches together with other researchers at Sony. Sony dog robot ("AIBO") uses a descendant of the opreating system. I joined KAME and IIJ in April 1998 (IIJ is a participant of KAME). At IIJ I help them deploy IPv6 network, and provide them with IPv6 and other advanced networking technologies based on KAME work.
HF:	There's the rumour that you never sleep and the only time that you don't read mail is when you're answering someone else's mail. Is that true? ;-)
I:	I got a couple of doppelgangers :-). Seriously, I do sleep every day! If you carefully check email timestamps, I almost never send emails between 3AM to 8AM, Japan time.
HF:	Do you work on other projects than KAME/IPv6? What's your favourite pastime, hobby, ...?
I:	My favorite area of hacking other than OS/network are like multilingualization and mobile devices. As a native Japanese speaker it is critical for me to have a good multilingualized software, so I work on it :-). I'm a developer of multilingual nvi, as well as citrus BSD locale framework. I used to collect digital cameras and shoot random pictures. These days I do not play with digital cameras, maybe I should do this again. Pastime... I love computers too much, it is a problem. I feel so comfortable when I'm typing :-). Other than computers, I like foods, both hot foods (like Indian, Thai) and non-hot foods. I love movies very much, DVDs are piling up in my room. Recently I saw lots of Korean movies. I highly recommend "Christmas in August". I'm an indoor guy:-)
HF:	Besides your work on bringing IPv6 into various BSDs, you are working on the OpenBSD project and are also a core member of the NetBSD projects. What's your jobs there, do you participate in these projects beyond merging the IPv6 code into the existing network stacks?
I:	I'm a developer for NetBSD, OpenBSD and FreeBSD. I don't think there's too many people with commit accesses to 3 BSDs, I believe more people should do this :-). Part of my task in KAME is to backporting fixes made by KAME into NetBSD and OpenBSD. So I do a lot of IPv6 integration work in OpenBSD. In the OpenBSD community, beyond IPv6 integration, I work on security issues in networking code, and sometimes other portions. There are lots of work to be done - it is amazing we still find bugs in BSD IPv4 code, even though it was implemented so long ago. I get so much help from OpenBSD folks, in term of security perspective. I always try to backport changes I made to a BSD, to other *BSDs. So, security changes I made on OpenBSD will get ported back to KAME tree, and other BSDs (I'm very sure that it will get ported back to NetBSD). I find good thing and bad thing in having multiple BSD codebases. Good thing is that we have projects with different goals, stimulating and encouraging each other. One negative thing are that manpower is splitted, code becomes diverged and becomes harder to port. I think I'm trying to solve part of the negative side.
HF:	What about FreeBSD, BSDi and Darwin, is there someone working on these projects like you work on NetBSD?
I:	FreeBSD and BSD/OS also integrate KAME. FreeBSD guys like Kris Kennaway and Hajime Umemoto helps us synchronize FreeBSD IPv6 code and KAME tree. For BSD/OS, since we cannot look at the source code, a BSDi guy is doing the integration, and we comment on the integration looking at beta releases. It seems that there are people working on KAME integration into Darwin/MacOS X. I really hope to see MacOS X shipped with IPv6 support, that will boost IPv6 userbase.
HF:	General acceptance of of IPv6 seems to be very low as people don't feel a pressing need to migrate from IPv4. When do you think will 25%, 50%, 100% of the internet sites talk IPv6?
I:	I'm not very sure, but if you are talking about "IPv6 enabled devices" UNIX boxes are becoming more and more IPv6 ready. If you install Solaris 8, it is IPv6 ready. I heard that next official release of Windows2K (2001?) will be IPv6 ready. People just need to enable it, or configure it, that's all. So I see obstacle in ISP side, and router vendors side. They have kind of deadlock situation: ISPs do not deploy IPv6 until routers get ready, and vendors do not ship routers until ISPs deploy it. I would like to change the situation by making code more available. I don't understand why people are keep using NAT, for me they are wasting their time! I had more than enough troubles with NAT, I really need simpler solution - that is, IPv6. Internet technology has to be much more stable, simple, and robust, unlike configurations with NAT. I believe people need to get informed better.
HF:	Where do you see applications of IPv6 that can speed up it's deployment, what's the "killer application"?
I:	To deploy IPv6, I believe we just need to make sure more products are IPv6 ready, and more number of ISPs are IPv6 ready (or at least their equipments are). This is what we do now. Some says that IPv4 was deployed because of WWW, and call WWW "the killer application". I feel the story is too simplified. IPv4 has been there even before WWW hit the street. There were lots of ISPs, there were university networks, IPv4 was deployed enough to hit the critical mass. Then, WWW appeared and IPv4 deployed to much more wider people. So, what I believe is we need to deploy IPv6 enough first, and then someone can come up with an application that is not possible with IPv4. I do not really have concrete image about how that application will look like. IPv6 can be used for much wider purposes, due to bigger addresses, including cellphones, lightbulbs, vehicles and household appliances, it will make a big difference. From implementation and specification point of view, IPv6 is ready to replace IPv4. I do everything I would do with IPv4, by IPv6. I don't have to deal with NAT, I don't have to worry about address space shortage any more, it's just better. We just need wider availability for it.
HF:	Is there anything left you'd like to tell our readers?
I:	If you have latest releases of NetBSD, OpenBSD or FreeBSD, you have IPv6 builtin, enabled by default. Try "telnet localhost" and you are using IPv6! The best way to learn about IPv6 is to actually use it, so please try it out, and do not hesitate to ask questions at users@ipv6.org. IMHO Japan is the most cool place for BSD geeks, and geeks in general. There are magazines dedicated to BSD, shops specialized in BSDs and UNIX, and coolest laptops and PDAs. Please visit Japan when you have a chance.