Stable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input.
For more information about how Stable Diffusion functions, please have a look at 🤗’s Stable Diffusion blog.
The Stable-Diffusion-v1-5 checkpoint was initialized with the weights of the Stable-Diffusion-v1-2
checkpoint and subsequently fine-tuned on 595k steps at resolution 512x512 on “laion-aesthetics v2 5+” and 10% dropping of the text-conditioning to improve classifier-free guidance sampling.
You can use this both with the 🧨Diffusers library and the RunwayML GitHub repository.
from modelscope.utils.constant import Tasks
from modelscope.pipelines import pipeline
import cv2
pipe = pipeline(task=Tasks.text_to_image_synthesis,
model='AI-ModelScope/stable-diffusion-v1.5-no-safetensor',
model_revision='v1.0.0')
prompt = '飞流直下三千尺,油画'
output = pipe({'text': prompt})
cv2.imwrite('result.png', output['output_imgs'][0])
finetune_stable_diffusion_lora.py
, run_lora.sh
sh run_lora.sh
import os
from dataclasses import dataclass, field
import cv2
from modelscope.metainfo import Trainers
from modelscope.msdatasets import MsDataset
from modelscope.pipelines import pipeline
from modelscope.trainers import EpochBasedTrainer, build_trainer
from modelscope.trainers.training_args import TrainingArgs
from modelscope.utils.constant import DownloadMode, Tasks
# Load configuration file and dataset
@dataclass(init=False)
class StableDiffusionLoraArguments(TrainingArgs):
prompt: str = field(
default='dog', metadata={
'help': 'The pipeline prompt.',
})
training_args = StableDiffusionLoraArguments(
task='text-to-image-synthesis').parse_cli()
config, args = training_args.to_config()
if os.path.exists(args.train_dataset_name):
# Load local dataset
train_dataset = MsDataset.load(args.train_dataset_name)
validation_dataset = MsDataset.load(args.train_dataset_name)
else:
# Load online dataset
train_dataset = MsDataset.load(
args.train_dataset_name,
split='train',
subset_name=args.train_subset_name,
download_mode=DownloadMode.FORCE_REDOWNLOAD)
validation_dataset = train_dataset
def cfg_modify_fn(cfg):
if args.use_model_config:
cfg.merge_from_dict(config)
else:
cfg = config
cfg.train.lr_scheduler = {
'type': 'LambdaLR',
'lr_lambda': lambda _: 1,
'last_epoch': -1
}
return cfg
kwargs = dict(
model=training_args.model,
model_revision=args.model_revision,
work_dir=training_args.work_dir,
train_dataset=train_dataset,
eval_dataset=validation_dataset,
cfg_modify_fn=cfg_modify_fn)
# build trainer and training
trainer = build_trainer(name=Trainers.lora_diffusion, default_args=kwargs)
trainer.train()
# pipeline after training and save result
pipe = pipeline(
task=Tasks.text_to_image_synthesis,
model=training_args.model,
lora_dir=training_args.work_dir + '/output',
model_revision=args.model_revision)
output = pipe({'text': args.prompt})
# visualize the result on ipynb and save it
output
cv2.imwrite('./lora_result.png', output['output_imgs'][0])
PYTHONPATH=. torchrun finetune_stable_diffusion_lora.py \
--model 'AI-ModelScope/stable-diffusion-v1.5-no-safetensor' \
--model_revision 'v1.0.0' \
--prompt "a cat" \
--work_dir './tmp/lora_diffusion' \
--train_dataset_name 'modelscope/DreamBooth-Dataset-Finetuning-Examples' \
--train_subset_name 'cat' \
--max_epochs 100 \
--save_ckpt_strategy 'by_epoch' \
--logging_interval 1 \
--train.dataloader.workers_per_gpu 0 \
--evaluation.dataloader.workers_per_gpu 0 \
--train.optimizer.lr 1e-4 \
--use_model_config true
should get train a lora weights file before using
from modelscope.utils.constant import Tasks
from modelscope.pipelines import pipeline
import cv2
pipe = pipeline(task=Tasks.text_to_image_synthesis,
model='AI-ModelScope/stable-diffusion-v1.5-no-safetensor',
lora_dir='path/to/lora/weights',
model_revision='v1.0.0')
prompt = '飞流直下三千尺,油画'
output = pipe({'text': prompt})
cv2.imwrite('result.png', output['output_imgs'][0])
prompt is very import to get a better result
photo realistic, Cinematic, filmic, portrait
3d cartoon, Pixar style, Realistic-3d-Cartoon
anime, acg, cartoon, animation
oil painting, masterpiece, trend on artstation
sketch, masterpiece, trend on artstation
watercolor, water color
from diffusers import StableDiffusionPipeline
import torch
model_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)
pipe = pipe.to("cuda")
prompt = "a photo of an astronaut riding a horse on mars"
image = pipe(prompt).images[0]
image.save("astronaut_rides_horse.png")
For more detailed instructions, use-cases and examples in JAX follow the instructions here
Download the weights
Follow instructions here.
Developed by: Robin Rombach, Patrick Esser
Model type: Diffusion-based text-to-image generation model
Language(s): English
License: The CreativeML OpenRAIL M license is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based.
Model Description: This is a model that can be used to generate and modify images based on text prompts. It is a Latent Diffusion Model that uses a fixed, pretrained text encoder (CLIP ViT-L/14) as suggested in the Imagen paper.
Resources for more information: GitHub Repository, Paper.
Cite as:
@InProceedings{Rombach_2022_CVPR,
author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
title = {High-Resolution Image Synthesis With Latent Diffusion Models},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {10684-10695}
}
The model is intended for research purposes only. Possible research areas and
tasks include
Excluded uses are described below.
Note: This section is taken from the DALLE-MINI model card, but applies in the same way to Stable Diffusion v1.
The model should not be used to intentionally create or disseminate images that create hostile or alienating environments for people. This includes generating images that people would foreseeably find disturbing, distressing, or offensive; or content that propagates historical or current stereotypes.
The model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
Using the model to generate content that is cruel to individuals is a misuse of this model. This includes, but is not limited to:
While the capabilities of image generation models are impressive, they can also reinforce or exacerbate social biases.
Stable Diffusion v1 was trained on subsets of LAION-2B(en),
which consists of images that are primarily limited to English descriptions.
Texts and images from communities and cultures that use other languages are likely to be insufficiently accounted for.
This affects the overall output of the model, as white and western cultures are often set as the default. Further, the
ability of the model to generate content with non-English prompts is significantly worse than with English-language prompts.
The intended use of this model is with the Safety Checker in Diffusers.
This checker works by checking model outputs against known hard-coded NSFW concepts.
The concepts are intentionally hidden to reduce the likelihood of reverse-engineering this filter.
Specifically, the checker compares the class probability of harmful concepts in the embedding space of the CLIPTextModel
after generation of the images.
The concepts are passed into the model with the generated image and compared to a hand-engineered weight for each NSFW concept.
Training Data
The model developers used the following dataset for training the model:
Training Procedure
Stable Diffusion v1-5 is a latent diffusion model which combines an autoencoder with a diffusion model that is trained in the latent space of the autoencoder. During training,
Currently six Stable Diffusion checkpoints are provided, which were trained as follows.
stable-diffusion-v1-1
: 237,000 steps at resolution 256x256
on laion2B-en.
194,000 steps at resolution 512x512
on laion-high-resolution (170M examples from LAION-5B with resolution >= 1024x1024
).
stable-diffusion-v1-2
: Resumed from stable-diffusion-v1-1
.
515,000 steps at resolution 512x512
on “laion-improved-aesthetics” (a subset of laion2B-en,
filtered to images with an original size >= 512x512
, estimated aesthetics score > 5.0
, and an estimated watermark probability < 0.5
. The watermark estimate is from the LAION-5B metadata, the aesthetics score is estimated using an improved aesthetics estimator).
stable-diffusion-v1-3
: Resumed from stable-diffusion-v1-2
- 195,000 steps at resolution 512x512
on “laion-improved-aesthetics” and 10 % dropping of the text-conditioning to improve classifier-free guidance sampling.
stable-diffusion-v1-4
Resumed from stable-diffusion-v1-2
- 225,000 steps at resolution 512x512
on “laion-aesthetics v2 5+” and 10 % dropping of the text-conditioning to improve classifier-free guidance sampling.
stable-diffusion-v1-5
Resumed from stable-diffusion-v1-2
- 595,000 steps at resolution 512x512
on “laion-aesthetics v2 5+” and 10 % dropping of the text-conditioning to improve classifier-free guidance sampling.
stable-diffusion-inpainting
Resumed from stable-diffusion-v1-5
- then 440,000 steps of inpainting training at resolution 512x512 on “laion-aesthetics v2 5+” and 10% dropping of the text-conditioning. For inpainting, the UNet has 5 additional input channels (4 for the encoded masked-image and 1 for the mask itself) whose weights were zero-initialized after restoring the non-inpainting checkpoint. During training, we generate synthetic masks and in 25% mask everything.
Hardware: 32 x 8 x A100 GPUs
Optimizer: AdamW
Gradient Accumulations: 2
Batch: 32 x 8 x 2 x 4 = 2048
Learning rate: warmup to 0.0001 for 10,000 steps and then kept constant
Evaluations with different classifier-free guidance scales (1.5, 2.0, 3.0, 4.0,
5.0, 6.0, 7.0, 8.0) and 50 PNDM/PLMS sampling
steps show the relative improvements of the checkpoints:
Evaluated using 50 PLMS steps and 10000 random prompts from the COCO2017 validation set, evaluated at 512x512 resolution. Not optimized for FID scores.
Stable Diffusion v1 Estimated Emissions
Based on that information, we estimate the following CO2 emissions using the Machine Learning Impact calculator presented in Lacoste et al. (2019). The hardware, runtime, cloud provider, and compute region were utilized to estimate the carbon impact.
@InProceedings{Rombach_2022_CVPR,
author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
title = {High-Resolution Image Synthesis With Latent Diffusion Models},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022},
pages = {10684-10695}
}
This model card was written by: Robin Rombach and Patrick Esser and is based on the DALL-E Mini model card.