LoRA can Replace Time and Class Embeddings in Diffusion Probabilistic Models

We propose LoRA modules as a replacement for the time and class embeddings of the U-Net architecture for diffusion probabilistic models. Our experiments on CIFAR-10 show that a score network trained with LoRA achieves competitive FID scores while being more efficient in memory compared to a score network trained with time and class embeddings.