Automatic Processing in Real-Life Image and Video Applications

Photorealistic Redesign of Interior Surface

Interior Redesign 

We investigate methods for texture synthesis and texture re-rendering of indoor room scene images. The goal is to create a photorealistic redesign of interior spaces by replacing surface finishes with a new product based on a single room scene image. Specifically, we focus on automating this process to reduce manual input while enabling high-quality and easy-to-use experience. We also present a click-based interactive segmentation for indoor scenes, which allows the user to select an object or region within the scene in a few clicks. The goal for the click-based approach is to provide the user with a simple method to reduce the amount of input required for segmentation.


  1. J. He, K. Ziga, J. Bagchi, F. Zhu, “CNN Based Parameter Optimization for Texture Synthesis,” Electronic Imaging, Burlingame, CA, USA, Jan 2019. (Best Student Paper Award)

  2. C.-J. Tai, T. Liu, J. Bagchi, F. Zhu, and J. P. Allebach, “Interactive Segmentation For Indoor Scenes,” Electronic Imaging, vol. 2017, no. 10, pp. 51-59, Burlingame, California, USA, Jan 2017.

  3. T. Liu, C.-J. Tai, F. Zhu, J. Bagchi, and J. P. Allebach, “Texture re-rendering tool for re-mixing indoor scene images,” Electronic Imaging, vol. 2017, no. 10, pp. 86-92, Burlingame, CA, USA, Jan 2017.

  4. K. Ziga, J. Bagchi, J. P. Allebach, and F. Zhu, “Non-parametric texture synthesis using texture classification,” Electronic Imaging, vol. 2017, no. 17, pp. 136-141, Burlingame, CA, USA, Jan 2017.

Caregiver-Infant Touch Detection

Touch Detection 

We investigate the detectin of interaction in videos between two people, namely, a caregiver and an infant. A particular type of action in human interaction known as touch is described, as touch is a key social and emotional signal used by caregivers when interacting with their children. We propose an automatic touch event recognition method to determine the time when the caregiver touches the infant and label it as a touch event by analyzing the merging contours of the caregivers hands and the infants contour. We present two methods to track the position of the hands using a tracking based method and a learning based method. We propose two Grab- Cut based infant segmentation methods to obtain the contour of the infant. The proposed method has been tested on our dataset that recorded by our partner with different pairs of participates, and has showed promising results compared to human annotated data.


  1. Q. Chen, R. Abu-Zhaya, A. Seidl, F. Zhu, “CNN Based Touch Interaction Detection For Infant Speech Development,” Proceedings of IEEE International Conference on Multimedia Information Processing and Retrieval, San Jose, California, USA, Mar 2019. (invited paper)

  2. Q. Chen, F. Zhu, “Long Term Hand Tracking with Proposal Selection,” Proceedings of the IEEE International Conference on Multimedia Expo Workshop, San Diego, CA, USA, Jul 2018.

  3. Q. Chen, H. Li, and R. Abu-Zhaya, A. Seidl, F. Zhu, E. J. Delp, “Touch Event Recognition for Human Interaction,” Electronic Imaging, Vol. 2016, no. 11, pp. 1-6, San Francisco, USA, Jan 2016.