Simple FFMpeg Script for 2D to 3D Video Conversion

Thomas Judge

Feb 11, 2022 • 5 min read

3D Full SBS Video Generated from Script for Avatar 2 Trailer

I have been experimenting with 2D to 3D video conversion for some time now, going back several years. A while back, I found an approach that used a script with AVISynth+ and FFMpeg, but I found the process very cumbersome and required the installation of a mass of software.

I recently reverse engineered that script and instead created a Windows CMD file that uses FFMpeg only.

As the encoding process was quite CPU intensive, I further established how to configure FFMpeg to use my Nvidia graphics card to do the encoding (to h265).

The output is a full 3D SBS file that plays fine in the likes of BigScreen VR Player, Pigasus or SkyBox VR player.

It works really well for me at least, I'd be interested in feedback from others and further optimisations to the script.

At the moment it generates left eye and separate right eye files before combing the two. There should be a clever way of doing this with FFMPeg without necessarily having to generate the intermediate left and right eye files first.

The basic idea behind the script is to combine two stretched video frames that are separated in time in order to form the left and right eye view. Your brain does the hard work of turning this into a convincing 3D visualisation, it works surprisingly well - believe me.

Usage:

To use it, download the file at the link below and save it as "cv3DSBS.cmd":

https://raw.githubusercontent.com/threedguru/threedguru/main/cv3DSBS.cmd

To convert a video called "my2dmovie.mp4" into a 3D video called "my3dmovie.SBS.mp4" run the following command from Windows Terminal with the script and video in the same folder (assuming FFMpeg is installed and on your windows path). This also assumes that you have an NVidia graphics card that supports hvec hardware encoding (I have a laptop with an RTX 3060 card for example):

./cv3DSBS.cmd my2Dmovie.mp4 my3Dmovie 0.04

The script generates a ".SBS.mp4" output file, so it would create a file called "my3Dmovie.SBS.mp4" from the above example.

The third parameter (0.04) in the example requires some explanation. It is the time delay between the left and right eye, in this case 1/25 of a second. This gives me the best results. The time delay needs to be long enough for there to be some relatively significant movement between the frames and also needs to be equal to or larger than the framerate of the input video (if it is smaller than the frame rate of the video then the left and right eye would be allocated the same video frame - and so no 3D effect).

It will work with other video formats as input not just .mp4.

I have downloaded and used one of the 2D Avatar 2 trailers available from here as an example. The input video in this case is an mkv file:

.\cv3DSBS.cmd .\AvatarTheWayOfWater_Trailer3StrongHeart_HEVC_1920x800_420_10bit_PCM_AC3_20-thedigitaltheater.mkv AvatarTrailer 0.04

This generates the 3D SBS video file "AvatarTrailer.SBS.mp4", see below:

This is a good example of the script output and what is possible, it is a Full Format 3D Side by Side (SBS) video.

You can open the link below in the Oculus browser in your headset. If you are not reading this in the headset browser, you really just have to remember the six characters "4od2ns" and that it is on the streamable.com web site (with https) to access it:

https://streamable.com/4od2ns

Press the play triangle icon within the video image to get it going, maximise it to full screen and select "3D Side by Side" from the drop down box in the Oculus Browser's video playback window.

The Oculus browser is expecting this to be a half height SBS which is why it looks a little odd. The browser does not seem to give much flexibility in terms of playback options compared to media players like; Pigasus, SkyBox VR player or BigScreen.

Full media players, for example, allow you to change the format or at least the aspect ratio, zoom in and out and various others things, which gives a much better experience. However, using the Oculus browser is an easy, if not ideal, demo.

I have used FFMpeg 4.4.1 for this work, as I had an intermittent hardware encoding problem with a later version of FFMpeg, so I went back to this one as someone else had reported a regression in later versions. Support for the hardware encoding is compiled into this version of FFMpeg.

There are other parameters within the script that can be tweaked in terms of the 3D effect, etc. Also the options to FFMPeg can be tweaked to modify the encoding, quality etc. I have set the encoding to "hevc_nvenc", which means Nvidia hardware hevc (h265) quality encoding, but this can be experimented with to get the best results, you can also change bit rate etc.

Background and Previous Work:

It is not my intention to add any dramatically new science or engineering here or (more importantly) to engage in a fruitless discussion about whether it works or not. I have provided some sample output from which to judge. I am simply building on the work of others. It isn't just mine. Please do not disrespect it.

The original script was posted in 2010 (not by me). Others, as well as myself, have applied it since. There are multiple use cases not just converting feature films. The technique is based on a scientific principle called the Pulfrich effect. For further background see below:

This script removes the need for AVISynth, addresses the increased frame rate of modern videos and includes support for the latest high compression encoding techniques with hardware support from the latest NVidia video cards.