This autumn, by request of our client, we developed a video deshaking method that
works in a single pass, in streaming mode. I know there is a very nice Deshaker filter
for VirtualDub but it works in two passes: first it collects motion data, then plans
for whole movie and in the second pass performs actual changes to the video. When
your video processing pipeline is quite heavy, loaded with many filters or something
computationally expensive, doing the work twice or saving it into a huge lossless file
and then processing twice with Deshaker can be extremely time-consuming. So, there
might be a need for something quicker, working in one pass.
Here are a couple of examples showing our own deshaker in action (before/after) :
The second clip
is cut from a recent earthquake movie, our deshaker is able to
fight the earthquakes rather well. ;) When deshaking applied, we can see how that
girl actively shakes the boy to simulate an earthquake. This is something not so
apparent in the original footage.
This is how the deshaker thinks about it:
From motion analysis it takes the main motion vector and uses it to model global
motion, drawn in blue (only vertical motion is shown here). By looking at motion
of up to 50 frames (using a sliding window) it chooses a trajectory (green)
which mostly consists of straight lines,
as if the camera mostly stays still or moves steadily without acceleration.
Then it shifts each frame by the difference between new and original trajectories.
Which means it's only good for translational motion as in examples above, it does not
currently fix any in-frame distortions nor rotation. It also zooms a bit
(so you don't see black borders near the edges), and the zoom factor is constant,
because as Deshaker for VirtualDub demonstrated, variable zoom factor can be really
sickening.
Currently this one-pass deshaker is a part of our SDK (among with super resolution
resizer, auto-brightness and film dirt cleaner filters) that we offer to other
developers. Internally it uses motion information collected by either super resolution
or dirt cleaner filters. We're
certainly going to include the deshaker in Video Enhancer 2.0 coming soon.
There is also a possibility to release it as a VirtualDub plugin or a plugin
for some other hosts. Please let me know if you'd like to see it as a plugin.
RSS
