How To Build An Emulab

Hardware

Here is a list of the items that make up the hardware for Calvin's Emulab:

• n (Currently 20+) PCs (nodes). Ours have 64-bit AMD processors, but you could get by with 386 chips. Each PC needs multiple NICs with the number depending on the total number of PCs. For up to 16 nodes each node must have 4 NICs. For up to 24 nodes each node must have 5 NICs. On board NICs are fine, but should not be used (if our experience is general) as the control interface. In fact, the NIC that connects to the control switch must be an Intel Pro 10/100 M. M, not S. M!

• Two servers, boss and ops. We use ordinary PC type machines with large disk drives.

• A Debian Router configured to run mrouted.

• Two Cisco 2950T switches for the Experimental Network.

• A Cisco 3550 switch (on loan from Calvin Information Technology (CIT)) for the Control Network. The fact that this switch is on loan is due to the biggest mistake that we made in this project: thinking that an unmanaged switch could perform this function. The control switch must be able to do IGMP snooping, and that requires a serious Cisco switch. Even the 3550 (unless it is enhanced) requires the cooperation of mrouted running on the Debian Router.

• A RocketPort RM32-RJ45 serial panel.

• Three PC-8000 Ethernet Power Controllers from Digital Loggers.

• A large number of ethernet cables. Specifically, 6n, where n is the number of PCs plus a small number to connect the servers and the router.

• Colored electical tape in at least 6 colors. We use this tape to distinguish cables. Very important.

Software

The documentation provided by Utah contains 40 text files. Fortunately, you do not need to consult all of them. For example, adding-nodes.txt refers to the old way of doing things. For starters we can limit the list to the setup files.

In some cases, a clear order is required. This order is deducible from within the documents. We will cover those first.

• setup-ops.txt.

  Here are some tips:

  1. Most recently tested on FreeBSD 4.11 means use FreeBSD 4.11.

  2. It is extremely important that /share be on a separate file system as is stressed in the documentation.

  3. See our defs file for details on what specific information you need to put in this critical file.

• setup-boss.txt. After OPS this is straightforward.

• setup-db.txt. At the heart of this enterprise is the tbdb database. Knowling how to manipulate this database has proved very helpful in cases where things do not quite go as planned. For example, sometimes an experiment refuses to terminate (it goes into transition mode). In that case it must be removed from the database (here's how). Fortunately, much of the information is entered into the database during the setup process or from a GUI interface. A second example of the use of the database is in the addition of a new node (not the first). This is almost all done for you using a gui interface:

  The first part of setup-db.txt is straightforward, but here is a useful tip. Circle the commands as you encounter them, for example,

  echo 'update users set admin=1 where uid="\"' | mysql tbdb

  If you added page numbers, it would be a good idea to note how to find that command. You can never remember it and you can never find it. A colored paper clip would also be a good idea.

  At various points you are warned against editing files directly. Listen to this advice.

  

  Ah, but your're not done yet. You need to add two entries for the node, one in tiplines and one in outlets. More on that later.

  The second part of setup-db.txt deals with protecting the webdb script and determining which users have the ability to edit the database. >p> The third part of setup-db.txt deals with switches. This is a tough section, and you should know something about Cisco switches or have someone around who does. You start by using the GUI interface but the "new_type=1" warns you that you aren't coming back here. We chose "cisco1950-ios" for our type and proc (not "processor") in the node_types table, and "switch" as the class (duh!).

  For the interface types we followed the example in the notes changing only the number of ports and the model. Note! It really is "manufacturuer", do not try to fix the misspelling!

  We slavishly imitated the interface_capabilities and nodes entries for our switches.

  Since we do connect our experimental switches together, we added interfaces for each of them.

  Upon examining the entries under switch_stack_types and switch_stacks, I have discovered that we did not follow the directions very well. Some research will be required before I continue.

  In step 4, power controllers, we seem to have strayed a bit. We added nodes called "power1", "power2", and "power3" and gave them a new type, "EPC". In /usr/testbed/bin/power we added code to deal with our type or power controller. See power_hack.

• setup-nodes.txt represents a watershed. Intuitively, you sense that once you have your first node up and running, you have an Emulab. You cannot do very interesting experiments, but you can do experiments! And if you can get the first one, you can probably get the second, and so it will go. Unfortunately, it is at this exact point where the setup files might be just a bit clearer.

  First off, we got an inside tip from Hussam Nasir at Kentucky. On boss there is a directory called /tftpboot under which you will find three important README files, freebsd.newnode/boot/README, freebsd/boot/README, and frisbee/boot/README. These files tell you how to run a script called prepare that simply produced "gzip'ed" versions of files to speed the various boot processes. There is no reason not to perform this step first.

  Second, there comes a point as you follow the directions in setup-nodes.txt (the exact point is still debatable in my mind) where you must perform a subroutine call. So follow the directions until you arrive at the point labelled "#### Bringing up the first node". On the next page you encounter "#### Bringing up the rest of the nodes", but there is quite a bit to do in between, and you will find the directions in customize-image.txt. Skeptical? See the line near the middle of the page that reads, "If you have followed the Emulab setup directions and have added nodes to your testbed to the point where they are now in the "hwdown" experiment, they should already be in the admin MFS." What you do is slogin or connect by serial console, load an image on the node, customize the image to your situation (your keys etc.), and then save the image back on boss for future use on all your other machines. The directions are actually straightforward, and the good news is that it is really true that once you have one node the rest are really easy. We did find that it was best to add nodes one at a time to avoid confusion (the alternative is the write down all their MAC addresses).

Now for the files for which no order is (to my knowledge) specified specifically.

• setup-network.txt gives some general principles for setting up your network. Calvin's is much simpler than most. Our nodes get their IP addresses from boss through our Debian router. As the documentation points out, "normally, DHCP traffic is not forwarded through routers". The trick here was to set up DHCP Relay on the router. This particular text file confirmed a glaring error in our original thinking (overview.txt should have been sufficient warning). "You need to make sure that IGMP snooping is enabled on the control switch." Now we were trying to build our Emulab with an off-the-shelf unmangaged switch, and, as predicted, frisbee was swamped. That is part of a longer story which will be told later.

• setup-serial-and-power.txt explains how to provide console access to the nodes as well as setting up the power controllers. As for the latter, the Digital Loggers controllers provided a convenient web interface, and setting them up was one of the easier parts of the project.

  It is important to realize that, as is pointed out in the last paragraph of setup-nodes.txt, creating a new node is not completely accomplished with the gui interface. See Adding PC19 for details.