Text to Image With ControlNet
Generate images based on depth or normal information obtained from the Unity camera
This makes it possible to use image generation in areas related to the game system where geometry is important.
Note: ControlNet requires a large amount of VRAM. If you are unable to generate images due to insufficient VRAM, you may need to reduce the resolution.
For generate image with Depth to Image, attach the Text2ImgWithControlNet.cs component to the object
Uncheck the CullingMask UI checkbox.
If you do not do this, the UI element will also get the depth information of the nearest neighbor, so it will be affected by the following RawImage, etc.
It is assumed that the local server is up and running.
Please refer to the following for StartServer before use.
🖥️About local serverSet the base 3D models in the range of the camera
A parameter summary is shown below. Parameters that are adjusted frequently are bolded.
Prompt
Enter the words you want to generate in natural language
NegativePrompt
Input elements in natural language that you do not want to include in the generated image
Size
Width and hight of the generated image in pixels
Steps
Increase this parameter if the generated image looks blurry
Cfg Scale
Strength to follow prompts Input larger value to strongly follow the prompt.
Seed
Fixing the seed value to generate a similar image
In case of -1, the image will be random
Batch Count
The number of images to be generated.
Select 1 to 100 images.
Generated images will be stored in StreamingAssets/StableDiffusion. Please check images by date sort in the explorer.
Camera
Set the camera with setting up for getting depth
ControlType
You can choose Depth or Normal
Weight
You can adjust the intensity of ControlNet's influence.
Sampler
Changing the nuances of the generated image
Model
Base trained model See the "Switch Models/LoRA" to switch new model
Lora
Additional trained Lora to generate consitent specific pattern image
See the "Switch Models/LoRA" to switch new LoRA
If you select Lora, <lora:filename:1>will be added to the Prompt and LoRA will be available
Last updated