In the Summer of 2005 Professors James Griffioen and Zongming Fei of the University of Kentucky and I, David Laverell of Calvin College received an NSF grant to demonstrate whether or not it was possible for a school of Calvin's size and resources to build an Emulab.
Upon learning that we had received the grant Professor Griffioen travelled to Grand Rapids to meet with me and with several network professionals from Calvin Information Technology. The results of that first brainstorming session can be seen below:
The terms of the grant involved undergraduate research so a key step was hiring Calvin students to do the actual work. In alphabetical order Tim Brom, Phil Davis, Josh Holtrip, and Eric Knibbe contributed most significantly. We also had Hussam Nasir at Kentucky who turned out to be a most valuable resource. Brian Baas, Jeff Greenfield, and Chris Wierenga of CIT and Gary Draving, the Computer Science Department Lab Manager also contributed in significant ways.
A description of the hardware can be found by clicking here.
After installing FreeBSD 4.11 on the two main servers, boss, the web server, and ops, users.emulab.calvin.edu, we spent a lot of time working on positioning them. Our first attempt was to place them behind a SuSE firewall. This led to many problems. boss is the DHCP server for the nodes, and this configuration led to those nodes having to contact boss at its secondary IP address. No good at all! Before too long we adopted Kentucky's approach and replaced the firewall with a Debian router. This meant that DHCP traffic had to go through that router so it had to do DHCP relay (an easy but interesting configuration).
After adding the rest of the hardware into our configuration, it was time to start installing the software. Our use of the text files provided by the University of Utah is documented in two places, under the Calvin Wiki and under what I think of as my commentary on the documentation.
Once you get a couple of nodes up you have the excitement of running some simple experiments, but when got up to nine nodes in the Spring of 2006 we started to encounter a problem that plagued us until December of that year! Emulab uses a disk imaging tool called frisbee (one of the messages you see a lot is "playing frisbee"). When an experiment is terminated or swapped the nodes involved play frisbee, they boot into a minimal operating system and wait to be part of a new experiment which may require one of several different images. As we started running experiments with nine nodes we discovered that only two or three would make it through the frisbee process leaving us with six or seven useless nodes. This was a conceptual error on our part. One of the things that was to be different about the Calvin Emulab was that the control switch was to be an unmanaged switch. Had we read overview.txt carefully, we would have realized that the control switch has to be pretty sophisticated, it must do IGMP snooping. This allows the switch to identify ports that are not interested in the multicast traffic and not send datagrams their way. The difference in price was quite significant. CIT came to the rescue by lending us a Cisco 3550. This improved the situation, but it was still not good. Hussam Nasir was still amazed at how long it was taking our nodes to go through frisbee.
One more incremental improvement involved our realization that the previous improvement was simply due to a more powerful control switch, but it was still not doing IGMP snooping. To do that the switch requires the cooperation of a router running mrouted. Fortunately, mrouted is easy to install on a Debian box, so we took that step. Still things were not working well enough. At this point there was a lot of frustration, and it was looking as though the answer to the question of whether a school of Calvin's size and resources could build an emulab was going to be a resounding, "No!" Then Jim Griffioen had a suggestion. Perhaps the on-board NICs were up to the task of being on the control switch. Testing this theory required a major configuration, we had to buy Intel 10/100 M cards, install them (I will never tell how much money we paid for the original extra NICs), add a new PC type with a different configuration order, and reinstall all the new nodes. A lot of work, but it paid off. The improvement was dramatic. For the first time we felt we had a working Emulab.
Alas, we had not seen the end of this problem. The Emulab worked fine as long as only one or two experiments were running, but when the Advanced Computer Networks course started using it in earnest, things slowed to a crawl. At this time, Tim Brom, who was working on the Emulab and also taking the course, needed to recompile a kernel to allow multicasting. When he tried to load his kernel onto a node, it took about six and a half minutes, and thus timed out. This was so even if this was the only thing going on at the time. Tim began to suspect that the problem was the speed of the disk on boss. Using a separate machine, not boss, he used the atacontrol program, which is distributed with FreeBSD to see what DMA mode the disks were operating under, and found it was the wrong one. After modifying the 4.11 kernel, Tim used bonnie++, a disk benchmark program, to exercise the disk over Christmas vacation to make sure he had not broken anything. After all this, Tim put the change into effect on boss. The speedup in the frisbee process was about ten. And that is the state of things today, June 7, 2007.