MasaCtrl: Tuning-free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

MasaCtrl enables performing various consistent non-rigid image synthesis and editing without fine-tuning and optimization.

模型描述

MasaCtrl，一个针对非刚性图像合成与编辑的免训练方法。主要方法是结合源图像的内容和文本题词、附加控制生成的布局来生成期望的图像。

a tuning-free method for non-rigid consistent image synthesis and editing. The key idea is to combine the contents from the source image and the layout synthesized from text prompt and additional controls into the desired synthesized or edited image, with Mutual Self-Attention Control.

(a) source; (b) reconstructed_source; (c) without masactrl; (d) masactrl
a photo of a running corgi

(a) source; (b) reconstructed_source; (c) without masactrl; (d) masactrl
a photo of a person, black t-shirt, raising right hand

期望模型使用方式以及使用范围

如何使用

代码范例

import cv2
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
from modelscope.utils.test_utils import test_level

prompts = [
    "",                           # source prompt
    "a photo of a running corgi"  # target prompt
]
output_image_path = './result.png'
img = 'https://public-vigen-video.oss-cn-shanghai.aliyuncs.com/public/ModelScope/test/images/corgi.jpg'
input = {'img': img, 'prompts': prompts}

pipe = pipeline(Tasks.image_editing,
    model='damo/cv_masactrl_image-editing')

output = pipe(input)['output_img']
cv2.imwrite(output_image_path, output)
print('pipeline: the output image path is {}'.format(output_image_path))

相关论文以及引用信息

@misc{cao2023masactrl,
      title={MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing}, 
      author={Mingdeng Cao and Xintao Wang and Zhongang Qi and Ying Shan and Xiaohu Qie and Yinqiang Zheng},
      year={2023},
      eprint={2304.08465},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
    }

Clone with HTTP

 git clone https://www.modelscope.cn/damo/cv_masactrl_image-editing.git