Content Aware Video Coding

Texture-Based Video Coding

Texture Model 

In recent years, there has been a growing interest in developing novel techniques for increasing the coding efficiency of video compression methods. One approach is to use texture and motion models of the content in a scene. Based on these models parts of the video frame are not coded or “skipped” by a classical motion compensated coder. The models are then used at the decoder to reconstruct the missing or skipped regions. We propose several spatial-texture models for video coding. We investigate several texture features in combination with two segmentation strategies in order to detect texture regions in a video sequence. These detected areas are not encoded using motion compensated coding. The model parameters are sent to the decoder as side information. After the decoding process, frame reconstruction is done by inserting the skipped texture areas into the decoded frames.

Publication:

  1. M. Bosch, F. Zhu and E. J. Delp, “Segmentation Based Video Compression Using Texture and Motion Models,” IEEE Journal of Selected Topics in Signal Processing, vol. 5, no. 7, pp. 1366-1377, Nov 2011. DOI:10.1109/JSTSP.2011.2164779

  2. M. Bosch, F. Zhu, and E. J. Delp, “Perceptual quality evaluation for texture and motion based video coding,” Proceedings of the IEEE International Conference on Image Processing, pp. 2285-2288, Cairo, Egypt, Nov 2009. DOI:10.1109/ICIP.2009.5414416

  3. M. Bosch, F. Zhu, and E. J. Delp, “An Overview of Texture and Motion based Video Coding at Purdue University,” Proceedings of the 27th Picture Coding Symposium, pp. 1-4, Chicago, USA, May 2009. DOI:10.1109/PCS.2009.5167388

  4. M. Bosch, F. Zhu, and E. J. Delp, “Models for texture based video coding,” Proceedings of International Workshop on Local and Non-Local Approximation in Image Processing, Lausanne, Switzerland, Aug 2008.

  5. M. Bosch, F. Zhu, and E. J. Delp, “Spatial Texture Models for Video Compression,” Proceedings of IEEE International Conference on Image Processing, pp. I-93-I-96, San Antonio, USA, Sep 2007. DOI:10.1109/ICIP.2007.4378899

  6. F. Zhu, K. Ng, G. Abdollahian, and E. J. Delp, “Spatial and Temporal Models for Texture-Based Video Coding,” Proceedings of SPIE 6508, Video Communications and Image Processing 2007, pp. 650806-650806-10, San Jose, USA, Jan 2007. DOI:10.1117/12.705068

Motion-Based Video Coding

Motion Model 

Using similar approach to texture-based video coding, we consider motion models based on human visual motion perception. We describe a motion classification model to separate foreground objects containing noticeable motion from the background. This motion model is then used in the encoder to again allow regions to be skipped and not coded using a motion compensated encoder. Our results indicate significant increase in terms of coding efficiency in comparison to the spatial texture-based methods.

Publication:

  1. M. Bosch, F. Zhu, and E. J. Delp, “Video Coding Using Motion Classification,” Proceedings of the IEEE International Conference on Image Processing, pp. 1588-1591, San Diego, USA, Oct 2008. DOI:10.1109/ICIP.2008.4712073

Deep Learning Based Approaches

Texture CNN 

There has been a growing interest in using different approaches to improve the coding efficiency of modern video codec in recent years as demand for web-based video consumption increases. We propose a model-based approach that uses texture analysis/synthesis to reconstruct blocks in texture regions of a video to achieve potential coding gains using the AV1 codec developed by the Alliance for Open Media (AOM). The proposed method uses convolutional neural networks to extract texture regions in a frame, which are then reconstructed using a global motion model. Our preliminary results show an increase in coding efficiency while maintaining satisfactory visual quality.

Publication:

  1. C. Fu, D. Chen, E. J. Delp, Z. Liu, F. Zhu, “Texture Segmentation Based Video Compression Using Convolutional Neural Networks,” Electronic Imaging, Burlingame, CA, USA, Jan 2018. arXiv:1802.02992

  2. Di Chen, Chichen Fu, Zoe Liu and Fengqing Zhu, “AV1 Video Coding Using Texture Analysis With Convolutional Neural Networks”, arXiv:1804.09291, 2018. arXiv:1804.09291

  3. D. Chen, Q. Chen, F. Zhu, “Pixel-Level Texture Segmentation Based AV1 Video Compression,” Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, Brighton, UK, May 2019. DOI:10.1109/ICASSP.2019.8682641

  4. D. Ding, M. Zhan, D. Chen, Q. Chen, Z. Liu, F. Zhu, “Advances In video Compression System Using Deep Neural Network: A Review and Case Studies,” Proceedings of the IEEE, Mar 2021. arXiv:2101.06341, website