Combined Object Segmentation and Tracking
Many object tracking algorithms implicitly or explicitly use a coarse segmentation of each target object using a simple mask such as a rectangle or an ellipse. This coarse segmentation results in several problems in tracking. If the mask is too large, extra background pixels will be included causing target drift.
If the mask is too small, part of the target is not being tracked resulting in reduced robustness against partial occlusion. Our research has focused on how to efficiently include segmentation as part of tracking in order to avoid these problems.
The basic model we use for each target is a multivariate Gaussian probability distribution. To determine where a target is located we need to compute the likelihood of the target being present at various candidate locations. At each candidate location we first use the marginal distribution of the brightness of each pixel to perform a soft segmentation. This segmentation is then used to modify the covariance matrix of the Gaussian distribution so that only pixels which are part of the target are considered. The joint probability is computed and stored. Finally, we use Bayesian inference to determine the most likely target location.
Once the target location is determined we need to update the probability model for the target. This is done using a modified version of the PPCA (Probabilistic Principal Component Analysis) algorithm which takes into account the probability that each pixel was generated by the target.
Results for several tracking sequences are given in the video section below. Combined tracking and segmentation significantly outperforms fixed segmentation.
• Chad Aeschliman
• Johnny Park
• Two people interacting with partial occlusion [wmv]
• Three people interacting with significant illumination changes [wmv]
• Outdoors low-resolution person and a car [wmv]
• A large number of people in a train station [wmv]
C. Aeschliman, J. Park, A. C. Kak, "A Probabilistic Framework for Joint Segmentation and Tracking," Computer Vision and Pattern Recognition, 2010 [pdf]
