Homework 2 (Due Nov-14-04) |
Implement and experiment with a Skin-Color Tracker. . The input is a video recording. The output is a sequence of binary images of the same dimension. Your program will set the pixels of the output image sequence to 1 at skin regions, and set to 0 everywhere else. Here are example matlab functions you can use. If you don't fully understand the matlab functions, use the matlab help feature. (for instance: help aviread). You are also welcome to use another programming language, but then we can't help you that much with the homework.
Here is the "recipe" (email me if it's too confusing, I just typed this in very fast...:)
You can display that image with: image(im); axis image;
Now select a "training set" of skin pixels. You can do that with: skin_mask = roipoly; (and select with the mouse a skin region)
To get a list of pixel indices of the mask area use: skin_inds = find(skin_mask>0);
For better performance mark several training sets across different subjects.
Use all those labeled skin pixels to estimate a multivariate Gaussian (as
discussed in class). First build a matrix that contains the RGB values
of the labeled skin pixels:
skin_R = double(im(:,:,1)); skin_G = double(im(:,:,2)); skin_B = double(im(:,:,3));
skin_data_rgb = [skin_R(skin_inds),skin_G(skin_inds),skin_B(skin_inds)]; %
that's the skin data
all_data_rgb = [skin_R(:),skin_G(:),skin_B(:)]; % that's the entire image data
To calculate the mean and the covariance of the skin Gaussian, use the
matlab function:
skin_MN = mean(skin_data_rgb); skin_CV = cov(skin_data_rgb);
Now you are going to compute for the entire image (and successive images in the video) the probabilities that a pixel contains a skin region.
Write a matlab function P = gaussdensity(all_data_rgb,MN,CV);
If the input (all_data_rgb) is a Nx3 matrix, the output is a Nx1 matrix of
probability values.
Use following formula for the gaussian:
If you do it in a smart way in matlab, you can fully vectorize it (no loops,
just matrix inputs and matrix outputs).
You can reshape the result into a 2D layer with:
[rows,cols,d] = size(im); L1 = reshape(P,rows,cols);.
You can do the same for a "background layer" L2 (using a different gaussian trained on background pixles). The last layer should be an "outlier layer" that has constant probability for each pixel. For example L3 = ones(rows,cols)/256;.
The final output of your algorithm should be normalized layers (posteriories). You get them in normalizing the values of each layer such that they add up to 1 for each pixel location (L1(x,y)+L2(x,y)+L3(x,y)=1). Do that with S = L1+L2+L3; L1 = L1./S; L2 = L2./S; L3 = L3./S; (Assuming uniform prior probabilities, those values could be interpreted as "posteriori probabilities": P(skin | RGB). And the un-normalized L1 values as "conditional density": p(RGB | skin).)
You can visualize your results in generating a video of the skin layer (L1). Use the matlab function: avifile
Play with different color spaces (RGB vs HSV vs HS (with H on a circle)). Compute and plot those layers for your face and 2 other faces. Converting a RGB image into a HSV image can be done with: rgb2hsv
The "HSV on a circle" representation can be done with: S*[cos(2*pi*H),sin(2*pi*H)]; (if this doesn't make sense to you, let's discuss it in class again :)
that's all for now, gotta run. might clarify more up there. don't hesitate to email me with questions. I haven't tried my own code on this data, but it should work, if you feel this data is not good (maybe too much color quantized etc...) , let me know...
chris.bregler@nyu.edu