Often when people first hear about Video Enhancer and its main feature, super resolution video resizing, they say "No way, it cannot work! If information is lost in downsizing you cannot get it back." And when we're talking about total information in a video they are absolutely right. However when looking at particular frames... Here's an experiment for you. I captured a couple of phrases with my two revolutionary ASCII video cameras that capture text, unfortunately one camera has low resolution and only captures two out of three letters, and the other camera has a nice resolution but results are noisy. Here are a few frames captured by my ASCII cameras. Can you guess what were the original phrases?

Read more...

tags: super_resolution

This year we explored ways to accelerate and improve our Super Resolution engine, and overall research, development and testing took much more time than we anticipated but finally the new engine is available to our users. Today we're releasing it in a form of VirtualDub plugin. DirectShow filter is also ready, so after we change Video Enhancer to use the new version there will be an update for Video Enhancer too. Also, an AviSynth plugin will be released soon. Our advances in speed allow using it in video players to upsize videos to HD in real time on modern PCs and laptops.

Generally quality and speed depend on source video and your CPU'a abilities, but here's an example of changes between our old VirtualDub plugin (1.0) and the new one (2.0) on a particular file (panasonic4.avi) when upsizing from 960x540 to 1920x1080 on an old Quad Core 2.33 GHz:
Quality, in dB of Y-PSNR (higher is better):

version     old      new
fast mode  41.70    42.28
slow mode  42.07    42.73

version      old    new_rgb    new_yv12
fast mode   19.6     14.4        10.3
slow mode   28.7     19.5        14.0

To achieve these speed gains our SR implementation was rewritten from scratch to work block-wise instead of frame-wise. This way it doesn't need so much memory to store intermediate results and intermediate data never leaves CPU cache, avoiding spending so much time on memory loads and stores. Also, we learned to use SSE2 vector operations better. Unfortunately even in 2013 compilers still generally suck at vectorizing code, so it requires a lot of manual work.

tags: video_enhancer super_resolution

It's been a while since last Video Enhancer version was released. It's time to break the silence and reveal some news and plans.

Part of the passed year was spent on ScreenPressor-related projects for our corporate clients. But for last several months we've been cooking our new super resolution engine. It's not on GPU yet but we've found a way to accelerate it on CPU and use significantly less memory. Actually, reducing memory usage is the key to acceletarion: nowadays memory access is quite slow compared to computations, and if you compute everything locally in small chunks that fit into cache and don't store and read whole frames in memory several times, overall process gets much faster. For us that meant a complete rewrite of our super resolution engine, and this is what we did. While remaking the algorithm we had a chance to rethink many decisions baked into it. Additional CPU cycles freed by the acceleration could be spent to perform more computations and reach for higher quality. So we spent several months in research: what motion estimation method works best for our SR? Shall we work in blocks of size 16, 8 or even 4 pixels wide (the latter meaning just 2x2 blocks of original image)? Which precision to use for each block size? What is the right way to combine new frame with accumulated information? For motion estimation we actually tried it all: using source code generation we generated 160 different implementations of motion estimation and measured how well they performed in terms of quality and speed. It turned out usual metrics for selecting best motion compensation methods in video codecs and other applications do not give the best results for super resolution: minimizing difference between compensated and current frame (which works best for compression) doesn't provide best quality in SR when compensated frame gets fused with input frame to produce the new upsized image. Then the fusion: is our old method really good or maybe it can be improved? We used machine learning techniques to find the best fusion function. And found that actually it depends a lot on source video: what works best for one video doesn't necessarily work well for another video. We chose a weighted average for a selected set of HD videos which means quality of the new SR engine should be higher on some videos and possibly a bit lower on some other videos, it won't be universally better.

Now what it all means in terms of releases. The new SR engine is in final testing stage now and it's a matter of days before it's released in Video Enhancer 1.9.9 and new version of our Super Resolution plugin for VirtualDub. Then it will also be released as AviSynth plugin, for the first time. And then finally it will come as Adobe After Effects (and Premiere Pro, probably) plugin. Some years ago we offered this Premiere Pro & After Effects plugin but it hadn't updated for too long and became obsolete, current versions of these Adobe hosts are 64-bit and cannot run our old 32-bit plugin.

Video Enhancer 1.9.9 will be a free update, and it's going to be the last 1.x version. Next release will be version 2.0 with completely different user interface, where all effects will be visible instantly. Imagine "Photoshop for video". Update to 2.0 will be free for those who purchased Video Enhancer 1.9.6 or later.

We hope everything mentioned above will see the light this year. VE 1.9.9 in September, plugins in October, VE 2.0 later. Update: oops, our scheduling sucks as usual. YUV support and thorough testing took much more time, the releases postponed until December.

tags: video_enhancer super_resolution

As you already know, super resolution is a method to upsample video which for each frame uses information from neighbour frames. I was asked many times: how many frames does it use? Well, our SR implementation is "streaming": frame in - frame out. Internally for each new frame it uses its own result for previous frame, i.e. to upsample frame N it uses upsampled frame N-1 for which frame N-2 was used, for which frame N-3 was used and so on. In this sense all previous frames are used to make the current one. However video is changing from frame to frame and as new information gets accumulated old information gets forgotten. And when processing frame 100 there is hardly anything left from frame 1. So how many frames are really used?

To answer this question we took some files from our video resize shootout and upsized them 2 times first the whole files, then starting from frame 20, then starting from frame 40 etc. Then we measured PSNR for each frame and looked at the charts.

Read more...

tags: super_resolution

Our super resolution implementation internally upscales video 2 times. If you ask Video Enhancer or Super Resolution plugin to upsize video 4 times using one instance of SR, it will upsize it 2x with super resolution approach and then 2x more using just a high quality image resampling method. For this reason we usually recommended to make a chain of 2 SR instances, each upsizing 2 times. This is what we did in our comparison. But are two instances really required?

To answer this question we took 10 videos from the comparison and upsized them 4 times using just one instance of SR. Here are the results in terms of PSNR:

Read more...

tags: super_resolution

We're very close to releasing the next version of Video Enhancer. Today we added a scene change detection algorithm into RGB version of super resolution engine. This feature solves an annoying problem of all previous versions of VE: when a scene changes in the video, first frames of the new scene contain some noise resulted from SR fusing current image with previous hi-def frame. Now VE detects a change and starts processing the scene from scratch, as if it was another video. Basically this means that first frame of new scene is upsized by interpolation and later frames are processed with SR.

Here's an example. This is a first frame of a scene:

VE 1.9.3, no scenes detection

VE 1.9.4, scenes detection on

In the left picture you can see the noise near a logo in upper-left corner as well as around people's shoulders and heads. In the right picture you can see this noise is gone.

tags: video_enhancer super_resolution