CSCI-UA.0202: Lab setup and tools

Overview

The main tools in this class are an x86 emulator, called QEMU, and a compiler toolchain (including assembler, linker, loader, C compiler, debugger), which will produce code (often a small-scale operating system) that will run on top of QEMU. These tools are included on the platform on which you will do your lab work. You have two choices for this platform:

1. A virtual devbox that we provide. Instructions
2. Machines provided by CIMS (The Courant Institute), on which you all have been given accounts. Instructions

The choice is up to you; there is no right answer. Each option has advantages and disadvantages. The advantage to the first option is that the lab will run completely on your own computer (and does not require Internet access, provided that you have the lab description locally). A disadvantage is that you have to do a bit more setup (which we describe and walk through below). A second disadvantage is that you will be interacting with two layers of virtualization (the host of the devbox, and then QEMU); this could be confusing. (This could also harm performance, but that will not be a concern for most of the labs.) The second option is the other way around: after your account is set up, you have to do very little setup, and there will be only one layer of virtualization. The disadvantage to this option is that you will be interacting with a computer in a lab somewhere, which requires either going to the lab, or working over ssh (which in turn requires Internet access).

There is a third option, but you're on your own if you choose it:

3. Your own machine, without virtualization. We cannot support this option, and your code will be graded on the two platforms above. Instructions

Option 1: The virtual devbox

One way to ensure that we are all using a uniform development environment (short of us all using the same machine) is for us to use identically configured virtual machines. You can think of a virtual machine as a way to run a particular operating system (in our case, an Ubuntu desktop) on top of another operating system (the one that controls your laptop or desktop).

To get our virtual devbox running, you should do the following:

1. Download and install on your computer (e.g., your desktop or laptop) a virtualization platform:
   • VirtualBox (free for Mac, Linux, Windows): Download page This is the recommended option.
   • VMware Player (free for Linux and Windows, registration required)
   • VMware Fusion
2. Download to your computer the virtual devbox image that we have set up for you. Note: the file is 1.5 GB, so the initial download may take some time.
3. Using the virtualization platform (for example, VirtualBox), import the image that you saved. You can leave all of the settings to their default values. This, too, may take some time.
4. Using the virtualization platform, start up the virtual machine. You should see an Ubuntu desktop come online.
5. Login to the machine. Username: cs202. Password: labs.
6. At this point, you have a machine that will act like an Ubuntu desktop, and you'll do your development on this machine.
7. If you are using VirtualBox, you will probably want to enlarge the default "monitor size" of the virtual machine. For this to work, the scaling feature must be enabled; this may require pressing a command key plus C, depending on your platform.

Notice the potential confusion from this arrangement. VirtualBox is creating a fake machine (with a fake x86 CPU). Ubuntu desktop is running on that fake machine (though neither Ubuntu desktop nor the applications (compiler, editor, browser) running on Ubuntu desktop are aware that they are running on a fake machine). The confusion enters because one of the applications that you are going to run on Ubuntu desktop is QEMU, and what does QEMU do? It creates another fake machine on which one can run an operating system. In our case, this fake machine will run the lab OSes (the mini operating systems and, perhaps, Linux). To recap: our lab OSes will run on the fake machine that QEMU provides, QEMU is running on Ubuntu desktop, Ubuntu desktop is the operating system for the fake machine that VirtualBox provides, and VirtualBox is running on your computer's operating system (known as the "host" operating system).

All of this is to say that there are actually two levels of virtualization in play. The two levels serve different purposes. The purpose of the lower is to create a uniform devbox. The purpose of the upper is to create the illusion of a physical machine so that you can develop, debug, and run the lab code. (This is a good example of how virtualization serves multiple purposes.)

A potentially interesting point is that although the two levels of virtualization serve different purposes, and although they are implemented with different software (VirtualBox and QEMU, respectively), they accomplish their function using similar technology. We discuss this technology in depth in advanced OS classes. In the meantime, welcome to The Matrix....

Option 2: CIMS machines

If you were registered for this class before Monday, January 26, then, thanks to the CIMS sysadmins, you have an account on machines administered by CIMS (short for Courant Institute of Mathematical Sciences, which is the home of the CS department). Thank you CIMS sysadmins!! If you added this class after January 26, and you have never had a CIMS account before (you can determine whether you had an account by going here), then please email our course's staff email alias (the address is on our home page). In this email, explain that you registered after January 26; in that case, we will close the loop between you and the CIMS sysadmins. If you were registered before January 26, and you cannot login, do not email us about that; we will probably drop such emails, since the instructions for this case are given below.

A few words about this account: it is independent of your NYU NetID/home account. This account will be valid during the semester, and will give you access to all of CIMS's computing resources.

If you have never had a CIMS account before (roughly half of you), then you have received an e-mail at your nyu.edu email address with setup instructions.

If you have had a CIMS account before (the other half of you), then you did not receive any kind of notification; you should simply login, following the instructions below. If you do not remember your password, or cannot login for other reasons, then go here.

There are two options for using CIMS computing resources. They are described below.

Computer lab

You can use the desktop machines in the Computer Labs in Rooms 229 and 230 Warren Weaver Hall. Here is more information about these Labs. You probably want the Linux machines. Type your CIMS username and password to login. After you login, you are ready to start the labs.

Remote Access

If you are not on campus, then you can access both the desktops and a collection of servers that CIMS maintains, using ssh. We describe the steps below. At a high level, you will be logging into a machine called access.cims.nyu.edu, and from there you will login to the CIMS machine of your choice. Please do not skip that second part, meaning run your lab on access.cims.nyu.edu; this will be viewed as antisocial.

1. Gain access to the CIMS machines through ssh. The Secure Shell (SSH) establishes an authenticated, encrypted session with a designated machine. You probably want to become comfortable with ssh, if you are not already. A basic guide for interacting with ssh is available here. For more in-depth information, check here.

   • For Linux and OS X users, open a terminal (xterm or via the Terminal program, etc), and type the following:

     $ ssh -X [username]@access.cims.nyu.edu 

     In this command, [username] is your CIMS username, and access.cims.nyu.edu is the name of CIMS's entrance server. The parameter "-X" arranges for graphics painted by the programs you run remotely to show up on your local computer.

   • For Windows users, there are several ssh clients. A decent one is PuTTY. We will assume that you are going to use this software.

     1. Download Xming and install it.
     2. Download PuTTY.
     3. Open PuTTY and input access.cims.nyu.edu as the hostname
     4. Select "Connection" → "SSH" → "X11" on the left panel, and check the "Enable X11 Forwarding" box.
     5. Click on "Open". After the terminal prompt, enter your username and password.

   For all users, if you have successfully logged in, you will see a banner like this:

   	############################################################################
   	#                                                                          #
   	#    CIMS Access Server                                                    #
   	#                                                                          #
   	#    ** Unauthorized access is prohibited by law! **                       #
   	#                                                                          #
   	#    Please do not run CPU-intensive jobs on this server.  For             #
   	#    information regarding appropriate systems on which to run such        #
   	#    processes, see:                                                       #
   	#                                                                          #
   	#           http://cims.nyu.edu/u/computeservers                           #
   	#                                                                          #
   	#    Users requiring a Solaris SPARC environment can login to              #
   	#    crunchy12.cims.nyu.edu                                                #
   	#                                                                          #
   	#    If you have any questions, please send mail to:                       #
   	#            helpdesk@cims.nyu.edu                                         #
   	#                                                                          #
   	############################################################################
   	

2. Consult this list to choose a server. Then, from access.cims.nyu.edu, type the following command:

   $ ssh -X [username]@[server].cims.nyu.edu

   Be sure to use the -X flag again. Once you log on, you can test to make sure X forwarding is working with the following command:

   $ xclock &

   (A window should pop up with a simple clock.)

   You should also see your home directory on any of the machines.

3. Optional but recommended: set up "ssh keys", which would remove some of the hassle of typing your password every time you ssh to a machine. For this, see the tutorials above, or consult one of the TAs.

You are now ready to begin the labs.

Option 3 (unsupported): Using your own machine, without virtualization

If you really want to, you can build and install the tools directly on your own machine. Howver, we cannot support this option, so you are on your own, for the most part. Furthermore, for the purposes of grading, your code must run on the two platforms above, so you should still test on one of them.

Below we give instructions for Unix and OS X computers. It should also be possible to get the development environment running under windows with the help of Cygwin. Install cygwin, and be sure to install the flex and bison packages (they are under the development header).

Step 1: Build and install the QEMU emulator

We recommend you use our (really, MIT's) patched version of QEMU instead of the stock version that may come with your distribution. To build your own patched version of QEMU:

1. Clone the QEMU git repository from the MIT 6.828 people:
   git clone https://github.com/geofft/qemu.git -b 6.828-1.7.0
2. You may need to install the SDL and pixman development libraries to get a graphical VGA window. On Debian/Ubuntu these are libsdl1.2-dev and libpixman-1-dev:
   sudo apt-get install libsdl1.2-dev libpixman-1-dev
3. Configure the source code
   
   1. Linux: ./configure --disable-kvm [--prefix=PFX] [--target-list="i386-softmmu x86_64-softmmu"]
   2. OS X: ./configure --disable-kvm --disable-sdl [--prefix=PFX] [--target-list="i386-softmmu x86_64-softmmu"] The prefix argument specifies where to install QEMU; without it QEMU will install to /usr/local by default. The target-list argument simply slims down the architectures QEMU will build support for.
4. Run make && make install This step may require privilege (sudo make install)

Any build problems are likely due to missing dependencies: be sure your system has the above libraries installed as well as normal development tools such as gcc, make, etc.

Step 2: Set up the compiler toolchain

$ objdump -i

$ gcc -m32 -print-libgcc-file-name

$ sudo apt-get install build-essential gcc-multilib gdb

GCCPREFIX=

export PATH=$PFX/bin:$PATH
export LD_LIBRARY_PATH=$PFX/lib:$LD_LIBRARY_PATH      # On OS X, use DYLD_LIBRARY_PATH

tar xjf gmp-5.0.2.tar.bz2
cd gmp-5.0.2
./configure --prefix=/usr/local--prefix=$PFX
make
make install             # This step may require privilege (sudo make install)
cd ..

tar xjf mpfr-3.0.1.tar.bz2
cd mpfr-3.0.1
./configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX
make
make install             # This step may require privilege (sudo make install)
cd ..

tar xzf mpc-0.9.tar.gz
cd mpc-0.9
./configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX
make
make install             # This step may require privilege (sudo make install)
cd ..


tar xjf binutils-2.21.1.tar.bz2
cd binutils-2.21.1
./configure --prefix=/usr/local--prefix=$PFX --target=i386-jos-elf --disable-werror
make
make install             # This step may require privilege (sudo make install)
cd ..

i386-jos-elf-objdump -i
# Should produce output like:
# BFD header file version (GNU Binutils) 2.21.1
# elf32-i386
#  (header little endian, data little endian)
#   i386...


tar xjf gcc-core-4.6.1.tar.bz2
cd gcc-4.6.1
mkdir build              # GCC will not compile correctly unless you build in a separate directory
cd build
../configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX --with-mpc=$PFX \
    --target=i386-jos-elf --disable-werror \
    --disable-libssp --disable-libmudflap --with-newlib \
    --without-headers --enable-languages=c
MAC OS X 10.7 "LION" NOTE: The default clang compiler on OS X 10.7
cannot build a working version of GCC. Use the following configure
line to work around the problem:
../configure --prefix=/usr/local--prefix=$PFX --with-gmp=$PFX --with-mpfr=$PFX --with-mpc=$PFX \
    --target=i386-jos-elf --disable-werror \
    --disable-libssp --disable-libmudflap --with-newlib \
    --without-headers --enable-languages=c \
    CC=/usr/bin/gcc-4.2 CPP=/usr/bin/cpp-4.2 \
make all-gcc
make install-gcc         # This step may require privilege (sudo make install-gcc)
make all-target-libgcc
make install-target-libgcc     # This step may require privilege (sudo make install-target-libgcc)
cd ../..

i386-jos-elf-gcc -v
# Should produce output like:
# Using built-in specs.
# COLLECT_GCC=i386-jos-elf-gcc
# COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/i386-jos-elf/4.6.1/lto-wrapper
# Target: i386-jos-elf


tar xjf gdb-7.3.1.tar.bz2
cd gdb-7.3.1
./configure --prefix=/usr/local--prefix=$PFX --target=i386-jos-elf --program-prefix=i386-jos-elf- \
    --disable-werror
make all
make install             # This step may require privilege (sudo make install)
cd ..