Enhanced 2D to 3D Algorithm - Discomfort Elimination
The base 2D to 3D conversion algorithm applied here in earlier work, takes two time delayed video frames and maps them to the left and right eye views.
This is both an extremely simple and effective approach. It relies on the Pulfrich effect. It does, however, require that there is relatively good correlation between the video frames selected for the left and right eye views. If this is not the case then the approach temporarily breaks down. This typically happens when the source video contains fast moving action that generates excessive change from video frame to video frame or where there is a wholesale change of scene.
These large scale frame to frame changes, translating to large discrepancies in the left and right eye views, both destroy the 3D Pulfrich effect and cause a brief feeling of discomfort or disorientation for the viewer.
The algorithmic enhancement described here aims to eliminate any sense of discomfort or disorientation due to rapid motion or scene changes. It does this by first detecting rapid motion and scene changes between frames. It then ensures that, for those instances, the same view for the left eye and right eye is kept, thereby eliminating the risk of disorientation as the view temporarily reverts to 2D. The video samples below provide a demonstration of the algorithm in action.
A further enhancement, involving tuning and motion compensation of the right eye frame compared to the left is described here.
The first video shows a 2D to 3D converted version of the "Wakanda Forever Trailer". It uses the Pulfrich base algorithm, but in addition a mask has been generated where rapid motion or scene changes are detected in the video. The mask shows up as a pink frame allocated to the right eye (in bottom half of the video). The format of this video and the videos below is 3D Half Top-Bottom (HTB). You can view this page in a VR Headset with a browser (such as the Meta Quest browser) to watch them in 3D. Make sure that you select the 3D Top/Bottom option to view them.
These are all relatively low resolution examples, due to being mangled by hosting on the streamable website, but I have also created a set of full resolution examples for download:
The next video shows the result of replacing the pink mask, generated for the right eye, with the same frame content as used for the left eye - effectively returning the video to 2D for those frames:
The next video shows the result using the original base Pulfrich algorithm with no motion or scene detection for comparison purposes:
As a further example, the video below shows the full algorithm, with motion and scene detection, applied to the 2D trailed for Avatar 2: The Way of Water:
The version of the video below was generated without motion and scene detection, for comparison purposes:
I have also assembled a collection of low resolution converted trailers as further examples here.
The 2D source trailers can be found at the links below:
May 9, 2022 at 1:50 PM