Homework 2 (Due Nov-14-04)

Implement and experiment with a Skin-Color Tracker.  .  The input is a video recording.  The output is a sequence of binary images of the same dimension.  Your program will set the pixels of the output image sequence to 1 at skin regions, and set to 0 everywhere else.  Here are example matlab functions you can use. If you don't fully understand the matlab functions, use the matlab help feature. (for instance: help aviread).  You are also welcome to use another programming language, but then we can't help you that much with the homework.

Here is the "recipe" (email me if it's too confusing, I just typed this in very fast...:)

• Download the compressed AVI or the cropped frames (saved as a zip file of jpegs)
• In matlab you can load a sequence of frames with the aviread function. For example load the first 20 frames with:  mov = aviread('subject01.avi',[1:20]);
• You can "digout" the first frame from the mov structure with: im = mov(1).cdata;
• You can display that image with: image(im); axis image;

• Now select a "training set" of skin pixels.  You can do that with: skin_mask = roipoly; (and select with the mouse a skin region)

• To get a list of pixel indices of the mask area use: skin_inds = find(skin_mask>0);

• For better performance mark several training sets across different subjects.

• Use all those labeled skin pixels to estimate a multivariate Gaussian (as discussed in class).  First build a matrix that contains the RGB values of the labeled skin pixels:
skin_R = double(im(:,:,1)); skin_G = double(im(:,:,2)); skin_B = double(im(:,:,3));
skin_data_rgb = [skin_R(skin_inds),skin_G(skin_inds),skin_B(skin_inds)]; % that's the skin data
all_data_rgb = [skin_R(:),skin_G(:),skin_B(:)]; % that's the entire image data

• To calculate the mean and the covariance of the skin Gaussian, use the matlab function:
skin_MN = mean(skin_data_rgb); skin_CV = cov(skin_data_rgb);

• Now you are going to compute for the entire image (and successive images in the video) the probabilities that a pixel contains a skin region.

• Write a matlab function P = gaussdensity(all_data_rgb,MN,CV);
If the input (all_data_rgb) is a Nx3 matrix, the output is a Nx1 matrix of probability values.
Use following formula for the gaussian:

If you do it in a smart way in matlab, you can fully vectorize it (no loops, just matrix inputs and matrix outputs).

• You can reshape the result into a 2D layer with:
[rows,cols,d] = size(im);  L1 = reshape(P,rows,cols);.

You can do the same for a "background layer" L2 (using a different gaussian trained on background pixles). The last layer should be an "outlier layer" that has constant probability for each pixel. For example L3 = ones(rows,cols)/256;.

The final output of your algorithm should be normalized layers (posteriories). You get them in normalizing the values of each layer such that they add up to 1 for each pixel location (L1(x,y)+L2(x,y)+L3(x,y)=1). Do that with S = L1+L2+L3; L1 = L1./S; L2 = L2./S; L3 = L3./S; (Assuming uniform prior probabilities, those values could be interpreted as "posteriori probabilities": P(skin | RGB). And the un-normalized L1 values as "conditional density": p(RGB | skin).)

• You can visualize your results in generating a video of the skin layer (L1).   Use the matlab function: avifile

• Play with different color spaces (RGB vs HSV vs HS (with H on a circle)). Compute and plot those layers for your face and 2 other faces.  Converting a RGB image into a HSV image can be done with: rgb2hsv

• The "HSV on a circle" representation can be done with: S*[cos(2*pi*H),sin(2*pi*H)];  (if this doesn't make sense to you, let's discuss it in class again :)

• If you are totally excited about this, you could already implement the next step.  Using the L1 matrix you can find the biggest "blob" of skin.  Use for example some smoothing of the L1 layer and then connected components, and then estimate the center of mass of that blob. That could be your estimate of the "head location". Keep repeating that computation for each succesive frame.

that's all for now, gotta run.  might clarify more up there. don't hesitate to email me with questions. I haven't tried my own code on this data, but it should work, if you feel this data is not good (maybe too much color quantized etc...) , let me know...

chris.bregler@nyu.edu