Scene and Motion Detection to Eliminate Discomfort
A simple and effective implementation of the Pulfrich 2D to 3D conversion algorithm takes two time delayed video frames and maps them to the left and right eye views.
However, such an implementation requires that there is good correlation between the video frames selected for the left and right eye views, otherwise the 3D effect stops working for those frames. Moreover, large scale frame to frame changes, that translate to large discrepancies in the left and right eye views, also cause a feeling of discomfort or disorientation for the viewer. This typically happens when the source video contains fast moving action, generating excessive change from video frame to video frame, or where there is a wholesale change of scene.
The algorithmic enhancement discussed below aims to eliminate any sense of discomfort or disorientation due to rapid motion or scene changes. It does this by:
- Detecting rapid motion and scene changes between frames.
- Ensuring that, for those frames impacted, the same view for the left eye and right eye is retained.
Retention of the same frame for the left and right eye, during rapid motion or scene changes, completely eliminates the risk of disorientation as the view temporarily reverts to a stretched 2D view for that period. The video samples below provide a demonstration of the algorithmic enhancement in action.
A further enhancement, involving tuning and motion compensation of the right eye frame compared to the left is described here.
The first video shows a 2D to 3D converted version of the "Wakanda Forever Trailer". It uses the Pulfrich base algorithm, but in addition a mask has been generated where rapid motion or scene changes are detected in the video. The mask shows up as a pink frame allocated to the right eye (in bottom half of the video). The format of this video and the videos below is 3D Half Top-Bottom (HTB). You can view this page in a VR Headset with a browser (such as the Meta Quest browser) to watch them in 3D. Make sure that you select the 3D Top/Bottom option to view them.
These are all relatively low resolution examples, due to being mangled by hosting on the streamable website, but I have also created a set of full resolution examples for download:
The next video shows the result of replacing the pink mask, generated for the right eye, with the same frame content as used for the left eye - effectively returning the video to 2D for those frames:
The next video shows the result using the original base Pulfrich algorithm with no motion or scene detection for comparison purposes:
As a further example, the video below shows the full algorithm, with motion and scene detection, applied to the 2D trailed for Avatar 2: The Way of Water:
The version of the video below was generated without motion and scene detection, for comparison purposes:
I have also assembled a collection of low resolution converted trailers as further examples here.
The 2D source trailers can be found at the links below:
May 9, 2022 at 1:50 PM