jbhuang0604
Joined 12 years ago
Karma
61
Submissions
26
Member since
Jul 1
https://jbhuang0604.github.io/
View on Hacker NewsRecent submissions
Comment · 3 years ago
Thank you!
Comment · 3 years ago
The UI part is basically a way to organize the user's intention. In the backend, we develop method for extracting "token maps" (i.e., which spatial regions correspond to specific words) and use region-based diffusion to achieve these localized editing results.
The second half of the video provides an overview of the method. https://www.youtube.com/watch?v=ihDbAUh0LXk
Comment · 3 years ago
Yes! I am particularly excited about this feature.
Comment · 3 years ago
Yup, it could be similar, but it mostly only works for very simple prompts (e.g., one subject in the image).
For example, in Figure 11 of the paper (https://arxiv.org/pdf/2304.06720.pdf), you can see that full-text "rustic cabin -> rustic orange cabin" does not turn the cabin orange.
For coloring, the core benefit of our method is that it allows precise color control. For example, it can generate colors with rare names (e.g., Plum Purple or Dodger Blue) or even particular RGB triplets that we cannot describe well with texts.
You can examples in Figure 4 here: https://arxiv.org/pdf/2304.06720.pdf
Comment · 3 years ago
Yes, if you go to the huggingface demo: https://huggingface.co/spaces/songweig/rich-text-to-image
You can find the segmentation information on the bottom-right of the rich-text generation result.
Comment · 3 years ago
Got it! Thanks for the feedback! This is definitely something we can improve.
Comment · 3 years ago
Thanks for the comment! Our method is model-agnostic. It can be easily adapted to any LLM (aka the text-encode) and any text-to-image models.
For example, the method was originally tested in Stable Diffusion 1.4. But we can easily apply it to Stable Diffusion-XL (or any finetuned model like ANIMAGINE-XL) even though the new model has a different text encoder and U-Net weights.
Comment · 3 years ago
Yes, I think the "footnote" showcases this well. You can use it to interactively explore your visual imagination.
Some examples here: https://youtu.be/ihDbAUh0LXk?si=i3LFfkDXIDKKvne3&t=91
Comment · 3 years ago
Thanks! BTW, we recently implemented the Automatic1111 extension so you can install and use it directly with A1111.
Comment · 3 years ago
We compared with three other prompt term weighting (aka font size) methods:
- Prompt-to-Prompt
- Parenthesis
- Repeating
Our results show that Prompt-to-Prompt would generate artifacts and other heuristic methods are not very effective.
You can see some examples in Figure 21, 22, 23. https://arxiv.org/pdf/2304.06720.pdf
Comment · 3 years ago
Yes, the particular font family was just a "label". We don't really interpret what that font style would look like.