InstantCharacter Personalize Image Characters with a Scalable Diffusion Transformer

Here is a nifty framework that lets you personalize characters in your images, using simple text prompts to make them do anything you want. For example, you can use a photo of Elon Musk to generate an image of him holding a chainsaw. InstantCharacter does this with a simple text prompt and an image. You can also get your character to wear something specific or maintain a certain pose.

This framework is based on these two clever approaches:
First, a scalable adapter module is developed to effectively parse character features and seamlessly interact with DiTs latent space. Second, a progressive three-stage training strategy is designed to adapt to our collected versatile dataset, enabling separated training for character consistency and text editability. By synergistically combining flexible adapter design and phased learning strategy, we enhance the general character customization capability while maximizing the preservation of the generative priors of the base DiT model.
A demo of this is already available on Hugging Face. You can change guidance scale, number of inference steps, and other settings.
[HT]