Text to Image With ControlNet

Generate images based on depth or normal information obtained from the Unity camera

Input	Depth	Normal

This makes it possible to use image generation in areas related to the game system where geometry is important.

Note: ControlNet requires a large amount of VRAM. If you are unable to generate images due to insufficient VRAM, you may need to reduce the resolution.

For generate image with Depth to Image, attach the Text2ImgWithControlNet.cs component to the object

Uncheck the CullingMask UI checkbox.

If you do not do this, the UI element will also get the depth information of the nearest neighbor, so it will be affected by the following RawImage, etc.

It is assumed that the local server is up and running.

Please refer to the following for StartServer before use.

🖥️About local server

Set the base 3D models in the range of the camera

camera range

place 3D model in the range of camera

A parameter summary is shown below. Parameters that are adjusted frequently are bolded.

Parameter	Summary
Prompt	Enter the words you want to generate in natural language
NegativePrompt	Input elements in natural language that you do not want to include in the generated image
Size	Width and hight of the generated image in pixels
Steps	Increase this parameter if the generated image looks blurry
Cfg Scale	Strength to follow prompts Input larger value to strongly follow the prompt.
Seed	Fixing the seed value to generate a similar image In case of -1, the image will be random
Batch Count	The number of images to be generated. Select 1 to 100 images. Generated images will be stored in StreamingAssets/StableDiffusion. Please check images by date sort in the explorer.
Camera	Set the camera with setting up for getting depth
ControlType	You can choose Depth or Normal
Weight	You can adjust the intensity of ControlNet's influence.
Sampler	Changing the nuances of the generated image
Model	Base trained model See the "Switch Models/LoRA" to switch new model
Lora	Additional trained Lora to generate consitent specific pattern image See the "Switch Models/LoRA" to switch new LoRA If you select Lora, <lora:filename:1>will be added to the Prompt and LoRA will be available