On the main page is a link to a mini-tutorial on how to load in a texture image into an array of packed r,g,b pixels.
As I said in class, I'm going to open things up a bit between now and the end of the semester. You can focus on one or both of two directions moving forward: (i) texturing, and (ii) animation. You can also choose one or more other topics if you check with me first. As always, your job is to implement the requisite technology, and then to create interesting original content that shows that technology in action.
I have to get your grades in by Dec 23. I am going to try to give you as much time as possible between now and then to get your projects completed. Please note that I am going to spend some time in class next Tuesday (Nov 30) looking on-line at select projects that students have implemented to date. If you want me to look at something in particular then, please let me know.
In these on-line notes I am going to be focusing on textures. Meanwhile, you can also go ahead and start to play with animation instead. All you really need in order to do that is to build a model that has rotatable joints (such as the swinging arm example we looked at in class earlier this semester), and then start using the cubic spline curves that you have already implemented in order to create time-varying joint angles for rotating the joints in your animatable figure. For example, you might want to try to build a simple two-legged human figure and implement a walk cycle. Next Tuesday I am going to talk more extensively about animation topics.
In class we went over texturing. First we covered texture mapping and antialiasing, with particular emphasis on MIP mapping. Then we covered procedural texturing, with a mini-lecture on the noise function and how to use it to make interesting textures. Here are some notes that follow that discussion.
At its most basic, texture mapping is quite simple. When you build a geometric mesh, you place parametric coordinates (u,v) at each vertex, in addition to (x,y,z) and (nx,ny,nz). Note that you have just "fattened" each vertex from six floating point numbers to eight floating point numbers.
For most of the types of geometric meshes that you have been implementing, it is very straighforward to assign a (u,v) value to each mesh vertex - you can just use the parametric coordinates that you used to build the shape (sphere, cylinder, bicubic patch, ...) in the the first place.
When you move through the rendering pipeline all the way to the pixel, the (u,v) value at each vertex becomes a value of (u,v) at each pixel to be shaded. At that point you can use this interpolated (u,v) to do a look-up into a stored texture image, from which you can retrieve a texture color (rt,gt,bt).
Note that (u,v) are not transformed by the matrix that transforms the location and normal of the vertex. But (u,v) are indeed linearly interpolated during scan conversion, along with all the other geometric data at each vertex.
If you do pure vertex shading (as we have done up until now), then you will only have (r,g,b) information at each pixel once you have done the z-buffer step. In this case, the only thing you can do with the texture color is to use it to modulate the color at that pixel:
r *= rt
g *= gt
b *= bt
On the other hand, you can defer shading until later in the pipeline, by interpolating the surface normal vector coordinates down to the pixel level, and doing the Phong algorithm calculation at each pixel. In this case, you can use the retrieved texture to modify any input parameter to the Phong shading algorithm. This includes any information about the color or location of light sources, as well as any material surface parameter such as the Ambient, Diffuse, or Specular color, or the Specular power, in addition to geometric data such as the surface normal.
If you interpolate surface normals between vertices rather than colors, and then do the Phong algorithm at the pixel level, then you are implementing a pixel shader. This can be much slower than doing a vertex shader, but it is also much more flexible and can produce more dramatic and varied surface effects. As I mentioned in class, pixel shaders are extensively used for feature films, when rendering time is not a critial issue, as well as in hardware-accelerated shaders enabled by GPUs (Graphics Processing Units), such as the nVidia GForce and ATI Radeon boards.
Antialiasing
As we discussed in class, you will generally get bad results if you simply use the (u,v) value at each pixel to do a look-up into your source texture image, because you will end up sampling improperly. The problem is that one image pixel can actually cover many texture source pixels, and what you ideally want is to perform an integral over the entire sub-area of the source texture image that is covered by each image pixel.
Doing a brute-force calculation of this area integral can be prohitibively expensive, so people have devised various methods to approximate this integral. The most commonly used even today is MIP mapping (where "MIP" stands for multim in parnum), developed by Lance Williams about 23 years ago. Lance adapted the power-of-two image pyramid strategy first developed by Tanimoto in the late 1970's for computer vision. This is one example of many in which computer vision algorithms for scene analysis have a parallel in computer graphics algorithms for scene synthesis.
Conversion to an image pyramid recursively proceeds by recursively halving the resolution of the texture image. At each stage of the recursion, every 2×2 pixels is replaced by a single pixel that contains the average value of those four texture pixels. The recursion stops when you are left with only a 1×1 image. It is simplest to begin with a source texture image that is a power of two in dimension, 256×256. This is converted into a pyramid of images:
For any pixel texture look-up, we look not just at the value (u,v) at that pixel, but also at the (u,v) at one or more neighboring pixels in the image to be rendered, in order to get an approximation of how much u and v change from one pixel to the next. The magnitude of this variation s is used to create a square shaped extent (u+s/2,v+s/2), over the texture image, which approximates the shape in the texture image that is covered by one pixel.
Now we can use this square shaped extent to do a tri-linearly interpolated look-up into the image pyramid, in which we linearly interpolate in the three dimensions of u, v, and scale. As I discussed in class, this will involve a total of eight accesses into the image pyramid (four each at two neighboring levels of the pyramid), and a total of 23-1, or seven, linear interpolation computations.
Procedural texture
Noise function
You can grab a copy of the source code for the noise function at: http://mrl.nyu.edu/~perlin/noise.
In class we went over an on-line tutorial about the noise function. You can review that tutorial at http://www.noisemachine.com/talk1/.
Using the noise function to make procedural textures
In addition to the examples at that tutorial, here is an on-line example of the use of noise function to vary surface normal: http://mrl.nyu.edu/~perlin/bumpy/ Feel free to look at the source code for class Sphere.java, in order to see how it all works.
The examples at that URL show procedural textures being used to vary the surface normal vector at each pixel, prior to performing the Phong shading algorithm at each pixel. Please note that this example does not use a real geometric model. Rather, the geometry of the spherical ball is faked by an image-based procedure, just for the purposes of the on-line demo.
In this on-line example, I first approximate the three partial derivatives of the noise function by taking differences in each dimension:
(noise(x+ε,y,z) - noise(x,y,z)) / ε
(noise(x,y+ε,z) - noise(x,y,z)) / ε
(noise(x,y,z+ε) - noise(x,y,z)) / ε
Then I subtract this vector-valued derivative function from the surface normal, to do bump mapping prior to performing the Phong shading algorithm. Bump mapping doesn't actually perturb the geometry of the surface, but rather fools the eye into thinking that the surface is perturbed, by changing the surface normals so that the surface responds to light the way an actual bumpy surface would.
If you choose to implement procedural textures, you shouldn't just duplicate the textures that I show in that example. Rather, you should play around with the technique, and try to create your own examples of procedural textures that vary Phong shading parameters or surface normal, or some combination, to create an interesting procedural texture.