DeOldify for Video Colorizaiton

English Version | 中文版本

Input a grayscale video and automatically output the corresponding color video.

Model description

DeOldify is a famous open source algorithm for automantic colorization. The model uses ResNet as encoder to build a network with UNet structure, and puts forward several different training versions, which has a good comprehensive performance in the aspects of effect, efficiency, robustness and so on.

The model here is the video model of DeOldify, which makes some optimization in reducing flicker. The video colorization process is realized using isolated image generation without any sort of temporal modeling tacked on.

Model structure

Usage

Given a sequence of grayscale video frames, color video frames will be generated. When providing color video frames, the result frames will be recolored.

How to use

With the ModelScope framework, you can use the colorization model through a simple Pipeline call.

Code example

from modelscope.outputs import OutputKeys
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

video = 'https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/public/ModelScope/test/videos/gray.mp4'
colorizer = pipeline(Tasks.video_colorization, model='damo/cv_unet_video-colorization')
result = colorizer(video)[OutputKeys.OUTPUT_VIDEO]

Limitations and possible deviations

• This model is the video model of DeOldify, which may be less colorful.
• Due to the limitations of training data and model structure, the algorithm may have some generation defects.

Evaluation results

This algorithm is tested on ImageNet，COCO-Stuff and CelebA-HQ.

Reference

@misc{deoldify,
    author    = {J. Antic},
    title     = {A deep learning based project for colorizing and restoring old images (and video!)},
    howpublished = "\url{https://github.com/jantic/DeOldify}",
}

DeOldify 视频上色

English Version | 中文版本

输入一段黑白视频，全自动地输出相对应的彩色视频。

模型描述

DeOldify 是上色领域比较有名的开源算法，模型利用 ResNet 作为 encoder 构建一个 UNet 结构的网络，并提出了多个不同的训练版本，在效果、效率、鲁棒性等等方面有良好的综合表现。

本模型为用于视频的版本，在减少视频中的闪烁方面做出了一定优化。视频上色过程是使用对每一帧的独立上色来实现，没有附加任何类型的时间建模。

模型结构

期望模型使用方式以及适用范围

本模型适用范围较广，给定一段任意的视频，都能生成上色后的彩色视频。如果输入的是彩色视频，将进行重上色。

如何使用

在 ModelScope 框架上，提供任意视频，即可以通过简单的 Pipeline 调用来使用视频上色模型。

代码范例

from modelscope.outputs import OutputKeys
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

video = 'https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/public/ModelScope/test/videos/gray.mp4'
colorizer = pipeline(Tasks.video_colorization, model='damo/cv_unet_video-colorization')
result = colorizer(video)[OutputKeys.OUTPUT_VIDEO]

模型局限性以及可能的偏差

• 本算法使用的是用于视频上色版本的 DeOldify 模型，在色彩鲜艳度上可能会稍逊一些。
• 因训练数据、模型结构等限制，算法存在一些生成瑕疵效果。

数据评估及结果

本算法主要在 ImageNet，COCO-Stuff 和 CelebA-HQ 上测试。

相关论文以及引用信息

@misc{deoldify,
    author    = {J. Antic},
    title     = {A deep learning based project for colorizing and restoring old images (and video!)},
    howpublished = "\url{https://github.com/jantic/DeOldify}",
}