Learning Hand Articulations by Hallucinating Heat Distribution

by | Aug 7, 2017

Authors: Chiho Choi, Sangpil Kim, Karthik Ramani
Proceedings of the IEEE International Conference on Computer Vision, 3104-3113


We propose a robust hand pose estimation method by learning hand articulations from depth features and auxiliary modality features. As an additional modality to depth data, we present a function of geometric properties on the surface of the hand described by heat diffusion. The proposed heat distribution descriptor is robust to identify the keypoints on the surface as it incorporates both the local geometry of the hand and global structural representation at multiple time scales. Along this line, we train our heat distribution network to learn the geometrically descriptive representations from the proposed descriptors with the fingertip position labels. Then the hallucination network is guided to mimic the intermediate responses of the heat distribution modality from a paired depth image. We use the resulting geometrically informed responses together with the discriminative depth features estimated from the depth network to regularize the angle parameters in the refinement network. To this end, we conduct extensive evaluations to validate that the proposed framework is powerful as it achieves state-of-the-art performance.

Please click here to download our paper.

Supplementary material is also available, click here to download.

Sangpil Kim

Sangpil Kim

Sangpil Kim is a Ph.D. student in the School of Computer Engineering at Purdue University. He is working on the deep learning algorithm and virtual reality. To be more specific, he develops the generative model, video segmentation, and hand pose estimation with a depth sensor. Currently, he is working on combining virtual reality and deep learning algorithm.