TÜBİTAK-2005: ENHANCE, "ENCODER ASSISTED VIDEO RESOLUTION ENHANCEMENT FOR COMPRESSED VIDEOS ", Işık University. Principal investigator: Hasan F. Ateş. 2009-2011.

Today, television and film industry is going digital very fast. The demand for high quality video and TV broadcasting, and the proliferation of  large screen LCD and plasma TV sets necessitate the production and transmission of spatially and temporally higher resolution (HR) content. The bandwidth required for high-definition (HD) broadcasting creates a bottleneck for the limited available resources of wired or wireless multimedia networks. The recently developed video coding standards (i.e., MPEG-4, H.264) and video quality enhancement techniques make it possible to broadcast in HD using such limited bandwidths.

Enhancing the resolution of a video coded at low-resolution (LR) is a problem that has attracted considerable attention from both academia and industry for quite some time. Temporal resolution enhancement, which is also called as frame rate (up) conversion (FRC), is important for smoother display of motion in video. For spatial resolution enhancement, super-resolution (SR) techniques are used that attempt to increase frame resolution by using available information from different frames of the same dynamic scene. While these technologies are important for developing better quality video decoders and TV displays, they are also difficult and expensive to implement due to the high computational complexity of the algorithms.

In this project, we propose an original video coding framework in which resolution enhancement is also an integral part of the encoder-decoder (codec). In this framework, a LR video is coded in such a way that the decoded information assists the decoder in performing SR and/or FRC as post-processing steps. Therefore, the encoder is assumed to know about these enhancement algorithms and is supposed to derive and code any necessary side information that can assist the decoder during SR and/or FRC of the LR video. The purpose of encoder assistance is two-folds: 1) If the video is intentionally reduced from HR to LR to save bitrate, the coded information could be helpful for better quality HR reconstruction at the decoder; 2) By performing computationally intensive parts of the enhancement algorithms (such as motion estimation) at the encoder side, high-quality SR and FRC could be achieved at the decoder at a significantly reduced level of complexity.

As part of this project, we plan to implement novel SR and FRC algorithms and related coding algorithms for the encoder-decoder interaction as described above. H.264 coding standard will be used and modified to incorporate the coding tools, parameters and other side information that are useful for resolution enhancement. Pre- and post-processing steps will be developed and optimized at the encoder and decoder side, respectively, to achieve highest quality HR video reconstructions. In designing the overall system, the goal is to provide an optimal trade-off between bitrate, reconstructed video quality, and the decoder complexity.

The proposed framework will be applied in two separate but related scenarios:

1.       HD video is downsampled (spatially and/or temporally) at the encoder and coded at a lower resolution. In this case, the encoder has access to the original HD video and could use this available information in assisting the decoder for better HD reconstruction. In [1], it has been shown that, for low-to-medium bitrates, coding at lower resolution could both reduce the required bitrate and improve the visual quality of the reconstructed video. 

2.       Standard definition (SD) (or any LR) content is provided to the encoder. This time, the encoder performs an extensive analysis and also applies SR and/or FRC algorithms to derive the best HD estimate of the video. Then, the encoder codes the essential parameters/side information that are learned through this initial analysis in a suitable format with as minimum additional bitrate as possible. This additional information is utilized by the decoder for fast and high-quality resolution enhancement of SD coded video.

Publications

Journal Papers

Conference Papers

§ B. Çizmeci, H.F. Ateş, "Occlusion Artifact Removal in Video Frame Rate Up-conversion",  accepted to SİU, Antalya, April 2009.