🔥Diffusers++🔥 is built on top of HuggingFace Diffusers, ensuring the inclusion of the latest state-of-the-art models related to image and video generation. 🔥Diffusers++🔥 is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🔥Diffusers++🔥 is a modular toolbox that supports both. Our library is designed with a focus on usability over performance, simple over easy, and customizability over abstractions.
Diffusers++ offers three core components:
-
Plus Pipelines and Plus Models include the latest advancements such as CHAMP, ELLA, and FIFO-Diffusion. We strive to incorporate the latest advances in the image and audio fields to ensure our library remains cutting-edge. Additionally, we offer state-of-the-art diffusion pipelines that can be run in inference with just a few lines of code.
-
Interchangeable noise schedulers for different diffusion speeds and output quality.
-
Pretrained models that can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems. Additionally, Plus Models replicate some of the latest advancements, including CHAMP and ELLA, to provide state-of-the-art performance.
We recommend installing Diffusers++ in a virtual environment from PyPI or Conda. For more details about installing PyTorch, please refer to their official documentation.
Currently, Diffusers++ can be installed through cloning the repository:
git clone https://github.com/ModelsLab/diffusers_plus_plus.git
cd diffusers_plus_plus
python -m pip install -ePlease refer to the How to use Stable Diffusion in Apple Silicon guide.
Generating outputs is super easy with Diffusers++. To generate an image from text, use the from_pretrained method to load any pretrained diffusion model (browse the Hub for 25.000+ checkpoints):
from diffusers import DiffusionPipeline
import torch
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipeline.to("cuda")
pipeline("An image of a squirrel in Picasso style").images[0]You can also dig into the models and schedulers toolbox to build your own diffusion system:
from diffusers import DDPMScheduler, UNet2DModel
from PIL import Image
import torch
scheduler = DDPMScheduler.from_pretrained("google/ddpm-cat-256")
model = UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda")
scheduler.set_timesteps(50)
sample_size = model.config.sample_size
noise = torch.randn((1, 3, sample_size, sample_size), device="cuda")
input = noise
for t in scheduler.timesteps:
with torch.no_grad():
noisy_residual = model(input, t).sample
prev_noisy_sample = scheduler.step(noisy_residual, t, input).prev_sample
input = prev_noisy_sample
image = (input / 2 + 0.5).clamp(0, 1)
image = image.cpu().permute(0, 2, 3, 1).numpy()[0]
image = Image.fromarray((image * 255).round().astype("uint8"))
imageCheck out the Quickstart to launch your diffusion journey today!
| Documentation | What can I learn? |
|---|---|
| Tutorial | A basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model. |
| Loading | Guides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers. |
| Pipelines for inference | Guides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library. |
| Optimization | Guides for how to optimize your diffusion model to run faster and consume less memory. |
| Training | Guides for how to train a diffusion model for different tasks with different training techniques. |
We ❤️ contributions from the open-source community! If you want to contribute to this library, please check out our Contribution guide. You can look out for issues you'd like to tackle to contribute to the library.
- See Good first issues for general opportunities to contribute.
- See New model/pipeline to contribute exciting new diffusion models/diffusion pipelines.
- See New scheduler.
- See Bug if something isn't working.
- See Documentation for improvements or additions to documentation.
- See Duplicate if this issue or pull request already exists.
- See Enhancement for new feature or request.
- See Help wanted if extra attention is needed.
- See Invalid if something doesn't seem right.
- See Question if further information is requested.
Also, say 👋 in our public Discord channel 
| Task | Pipeline | 🔥Diffusers++🔥 Hub |
|---|---|---|
| Generating Infinite Videos from Text(upcoming) | FIFO-Diffusion | dummy/dummy-pipeline |
| Text-to-Image | ELLA | dummy/dummy-pipeline |
| Parametric 3D Human Animation via Latent Diffusion | CHAMP | dummy/dummy-pipeline |
- https://github.com/ModelsLab/diffusers_plus_plus
- https://github.com/microsoft/TaskMatrix
- https://github.com/invoke-ai/InvokeAI
- https://github.com/apple/ml-stable-diffusion
- https://github.com/Sanster/lama-cleaner
- https://github.com/IDEA-Research/Grounded-Segment-Anything
- https://github.com/ashawkey/stable-dreamfusion
- https://github.com/deep-floyd/IF
- https://github.com/bentoml/BentoML
- https://github.com/bmaltais/kohya_ss
- +11,000 other amazing GitHub repositories 💪
Thank you for using us ❤️.
This library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today:
- @CompVis' latent diffusion models library, available here
- @hojonathanho original DDPM implementation, available here as well as the extremely useful translation into PyTorch by @pesser, available here
- @ermongroup's DDIM implementation, available here
- @yang-song's Score-VE and Score-VP implementations, available here
We work hard to add the latest text-to-image and text-to-video pipelines to ensure that our library remains cutting-edge and versatile. Our team is committed to integrating state-of-the-art models and providing a robust and user-friendly API for all users.
We also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, available here as well as @crowsonkb and @rromb for useful discussions and insights.
@misc{von-platen-etal-2022-diffusers,
author = {Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Dhruv Nair and Sayak Paul and William Berman and Yiyi Xu and Steven Liu and Thomas Wolf},
title = {Diffusers: State-of-the-art diffusion models},
year = {2022},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/huggingface/diffusers}}
}