Real-time Multimedia Applications

Headsets designed for Virtual Reality (VR), often referred to as Head Mounted Displays (HMDs), provide a significant means for individuals in the real world to engage with computer-generated virtual environments. One of the main challenges has been the manual annotation of real people facial expression parameters, known as Action Units. Synthetic data has therefore become a popular choice for model training. However, potential inaccuracies can arise due to discrepancies between the synthetic and real data distributions. Our research delves into this domain shift issue, bridging the gap between synthetic and real people data domains with domain adaptation methods. Publications:

1. X. Ji, J. Yang, J. Wei, Y. Huang, Q. Lin, J. P. Allebach, and F. Zhu, “VR Facial Expression Tracking via Action Unit Intensity Regression Model,” Proceedings of the Electronic Imaging, Jan. 2022.

2. X. Ji, J. Yang, J. Wei, Y. Huang, S. Zhang, Q. Lin, J. Allebach, F. Zhu, “Classifier Guided domain Adaptation for VR Facial Expression Tracking,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

3. J. Yang, X. Ji, J. Wei, Y. Huang, S. Zhang, Q. Lin, J. Allebach, F. Zhu, “VR Facial Expression Tracking Using Locally Linear Embedding,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

Video conferencing usage significantly increased during the pandemic period and is expected to keep popularity in the hybrid work. One of the key aspects of video conferencing experience is background blur and background replacement. In order to achieve a high-quality background blurring effect, accurate portrait segmentation and high-quality blurring effect are necessary. Existing methods have achieved high accuracy on portrait image segmentation, but valuable temporal information is often not fully leveraged. Motivated by this, our proposed method uses deep learning network combined with temporal information to perform high quality and consistent portrait video segmentation.

Publications:

1. W. Xu, Y. Shen, Q. Lin, J. Allebach, and F. Zhu, “Efficient real-time portrait video segmentation with temporal guidance”, Proceedings of the Electronic Imaging, California, USA, Jan 2022.

2. Y. Shen, W. Xu, Q. Lin, J. Allebach, and F. Zhu, “Depth assisted portrait video background blurring”, Proceedings of the Electronic Imaging, California, USA, Jan 2023.

3. Y. Shen, W. Xu, Q. Lin, J.P. Allebach, F. Zhu, “Real-Time End-to-End Portrait and In-Hand Object Segmentation with Background Fusion,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

4. W. Xu, Y. Shen, Q. Lin, J. Allebach, F. Zhu, “Exploiting Temporal Information in Real-Time Portrait Video Segmentation,” Proceedings of the ACM International Conference on Multimedia Workshop (ACMMM-W, HCMA), Ottawa, Canada, Oct 2023.