How to use AI frame interpolation to create “impossible” video shots

By Karen X. Cheng

Frame interpolation is a technique used to guess between frames. It’s often used to create artificial super slow-motion, but I thought it’d be fun to create “impossible” stop motion effects.

 

To try it yourself, you have two options:

Option 1 Frame Interpolation on Runway (works in your browser)

Option 2 Depth-Aware Video Interpolation (Windows only, open source)

 

Both options are free. Runway is easier to use, however they charge a $12/month subscription if you want to export your footage in HD.

 

Here are my tips for trying this technique:

 

Shooting

  1. Don’t move too much in between pictures. Frame interpolators do a good job when the frames are similar to each other, and not as well when there’s too much movement. If you notice artifacts, try reducing the amount of movement between pictures.

  2. Shots tend to work best when the background is fixed, so I recommend shooting with a tripod.

  3. After experimenting, I found my shots looked the most interesting when I shot on grass, sand, or dirt (rather than a smooth surface like concrete or a hardwood floor). This is because tiny movements cause the ground to move slightly, a nod to the practical effect of the person’s physical presence in the space.

Editing

  1. You can experiment with the amount of frame interpolation, e.g. how much time elapses in between frames. Both Runway and DAIN give you options to play with the time scale. For my shots, I’ll often try several different time scale options until I find the creative fit.

  2. When interpolating, you can work with either still images, or screengrabs from a video. Still images tend to be higher quality. However, if you use screen grabs from a video, you have the option to combine live action footage with stop motion.

  3. In your video editing software, you can combine multiple different shots to create a storyline, like I have in my example.

 

Because results can be unpredictable, you might want to bring your laptop to where you’re shooting so you can test out your footage right then and there, and reshoot if necessary. I edited this on my Dell XPS 13 and Dell Precision 5860.

Have fun experimenting!

Français