sambb - Flying steps [interpolated]

Since the first time I've seen a TV with integrated 60 fps interpoltion function I've wondered if it would be possible to do something like this with movie "sambb - Flying steps". If you've never seen it, here are link on youtube: one two. It was made 12 years ago long before 60fps came to youtube, so in order to make 30fps look smooth you had to use sampling in HLAE (mirv_sample_enable and mirv_sample_sps cvars). The logic as I see it: 30 fps and 90 sps meant that HLAE would take 3 times more screenshots and combine every three frames into one. And the problem is that this movie was made without any sampling at all and it looks quite laggy. So two months ago I started to search the way to do something about it.

The fisrt idea was to use some AI and I downloaded the free version of Dain-App, but it just failed to run with some internal error. Posting the report on their discord gave zero responce, so I've tried another similar app RIFE. I had to dig into console work, but in the end I've got the result and ... it was quite awful. As I realized later, it just randomly used blending in some scenes and optical flow in others. Weapons were flickering, picture went crazy a lot of times. I guess you need some correctly pretrained model for a heavily edited kz movie like this, but doing that was out of my skills.

The next thing I tried was optical flow in Adobe Premier Pro. You just need to set speed of the video to 50 percent and choose optical flow that will be applied to the whole video. The result is the same as I saw in AI, a lot of weapons/hands flickering. It happens because algorithm doesn't know which things are placed closer on the picture. Just to demostrate it I've uploaded it to youtube:

60 fps Premier Pro optical flow

So I've searched the plugin for Adobe After Effects to apply it only to the certain areas. The first try was Twixtor Pro. It has percent setting, that goes from 0 (pure blending, something like sampling in HLAE) to 100 (pure optical flow which is trying to predict next picture using so called motion vectors). Zero percent looked like original 30 fps, since half of the frames stayed original without any blending, so after some tests I've used default 70 percent. The hardest work here was to create dinamic masks that would cover hands and weapons and moved along with them. Moreover you cannot just remove the mask when it's not needed anymore cause it would cause a great distortion of the whole picture, so I had gradually change them into dot or move outside of the screen. And after all that work I met more nasty things - firstly jumper's style included a few sudden mouse turns that driven Twixtor crazy, and secondly algorithm had hard times on scenes that were heavily color corrected and contained very few colors (like kz_kzlv_wanderer bhop and final credits bhop). Plus despite using masks you still can notice a distortion near all moving objects like players model and weapons.

So I've tried another plugin called BCC optical flow. It also has percentage setting where 0 is blending, but 100 percent makes video unwatchable, so in the end I stayed at 60 percent that was quite ok. Some flickering was mentioned here too, so I've just used masks from my Twixtor experiment (one mask for weapon uses 0 percent and another inversed mask for the rest of the picture uses 60 percent). However it still doesn't look like 60 fps, I've uploaded it to youtube and mega:

60 fps BCC optical flow + masks

download from mega

Mega link is here cause youtube made quality MUCH worse, original 720p looks like 240p, cannot do anything about it.

And finally I've combined both methods, using Twixtor for most of the scenes and replacing it with BCC optical flow at moments of fast turns (6 places in total, half of them on kz_kzlv_wanderer) plus the whole final credits scene was rendered with BCC effect plus moving masks for the text, any Twixtor settings made this scene awful. So here's youtube and mega links as well (youtube has broken the quality again, so the videos are unlisted):

60 fps Twixtor Pro + BCC optical flow + masks

download from mega

I've challenge myself to do my best here, I've played with a lot of settings and made more then dozen of full 9 hours renders plus a lot of partial renders, however I still hope that it can be done better. Few days ago I've also tried FlowFrames software that includes Dain and RIFE algorithms, but it just failed to download pretrained models so the interpolation process wasn't initiated. Maybe someone will be more lucky on this.

really good read. The difference is huge, gj

@JocA the flickering of weapons is almost in every scene there. It's very good for fast turns I've mentioned, but I cannot apply it with mask. Tried to google it, but haven't found anything.

There's also optical flow in davinci resolve studio, might be worth adding it to the comparison

Thanks for the efforts. Tbh, I've always felt sort of uneasy watching the original version. So cool now