Real-time Multimedia Applications

Video conferencing has become an essential part of remote work. However, traditional conferencing setups with a single webcam cannot achieve immersive experience due to lacking the visual cues that convey depth and perspective. To address exsiting challenges, we introduce a method that interpolates novel views from a pair of input views to achieve free view experience. Improvements are established around the view interpolation method to achieve high fedilty and temporal consistent novel views.

Publications:

1. W. Xu, Y. Shen, Q. Lin, J. Allebach, and F. Zhu, “Pose Guided Portrait View Interpolation from Dual Cameras with a Long Baseline”, Proceedings of the 2024 IEEE 26th International Workshop on Multimedia Signal Processing (MMSP), Indana, USA, Oct 2024.

Video conferencing usage significantly increased during the pandemic period and is expected to keep popularity in the hybrid work. One of the key aspects of video conferencing experience is background blur and background replacement. In order to achieve a high-quality background blurring effect, accurate portrait segmentation and high-quality blurring effect are necessary. Existing methods have achieved high accuracy on portrait image segmentation, but valuable temporal information is often not fully leveraged. Motivated by this, our proposed method uses deep learning network combined with temporal information to perform high quality and consistent portrait video segmentation.

Publications:

1. Y. Shen, W. Xu, Q. Lin, J.P. Allebach, and F. Zhu, “GRIB: Combining Global Reception and Inductive Bias For Human Segmentation and Matting,” Proceedings of the 2024 IEEECVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)/, Washington, USA, July 2024.

2. W. Xu, Y. Shen, Q. Lin, J. Allebach, and F. Zhu, “Exploiting Temporal Information in Real-Time Portrait Video Segmentation,” Proceedings of the ACM International Conference on Multimedia Workshop (ACMMM-W, HCMA), Ottawa, Canada, Oct 2023.

3. Y. Shen, W. Xu, Q. Lin, J.P. Allebach, and F. Zhu, “Real-Time End-to-End Portrait and In-Hand Object Segmentation with Background Fusion,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

4. Y. Shen, W. Xu, Q. Lin, J. Allebach, and F. Zhu, “Depth assisted portrait video background blurring”, Proceedings of the Electronic Imaging, California, USA, Jan 2023.

5. W. Xu, Y. Shen, Q. Lin, J. Allebach, and F. Zhu, “Efficient real-time portrait video segmentation with temporal guidance”, Proceedings of the Electronic Imaging, California, USA, Jan 2022.

Denoising and super-resolution are two important tasks for video enhancement. In many real-world applications such as video conferencing, the input video obtained from a webcam or external camera is often noisy. Directly applying an SR network to noisy video will significantly undermine the performance of the SR and yield unwanted artifacts. There are very few works that target both tasks simultaneously, which motivates us to combine the two process into a joint architecture.

Publications:

1. Y. Huang, T. Wang, Q. Lin, J. P. Allebach, and F. Zhu, “Efficient Joint Video Denoising and Super-Resolution,” Proceedings of the 2023 IEEE International Conference on Image Processing (ICIP), Kuala Lumpur, Malaysia, Oct 2023.

Headsets designed for Virtual Reality (VR), often referred to as Head Mounted Displays (HMDs), provide a significant means for individuals in the real world to engage with computer-generated virtual environments. One of the main challenges has been the manual annotation of real people facial expression parameters, known as Action Units. Synthetic data has therefore become a popular choice for model training. However, potential inaccuracies can arise due to discrepancies between the synthetic and real data distributions. Our research delves into this domain shift issue, bridging the gap between synthetic and real people data domains with domain adaptation methods.

Publications:

1. X. Ji, J. Yang, J. Wei, Y. Huang, S. Zhang, Q. Lin, J. Allebach, and F. Zhu, “Classifier Guided domain Adaptation for VR Facial Expression Tracking,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

2. J. Yang, X. Ji, J. Wei, Y. Huang, S. Zhang, Q. Lin, J. Allebach, and F. Zhu, “VR Facial Expression Tracking Using Locally Linear Embedding,” Proceedings of the IEEE International Conference on Multimedia and Expo Workshop (ICME-W), Brisbane, Australia, Jul 2023.

3. X. Ji, J. Yang, J. Wei, Y. Huang, Q. Lin, J. P. Allebach, and F. Zhu, “VR Facial Expression Tracking via Action Unit Intensity Regression Model,” Proceedings of the Electronic Imaging, California, USA, Jan 2022.