Text to Image With ControlNet

Generate images based on depth or normal information obtained from the Unity camera

Input

Depth

Normal

This makes it possible to use image generation in areas related to the game system where geometry is important.

Note: ControlNet requires a large amount of VRAM. If you are unable to generate images due to insufficient VRAM, you may need to reduce the resolution.

For generate image with Depth to Image, attach the Text2ImgWithControlNet.cs component to the object

Uncheck the CullingMask UI checkbox.

If you do not do this, the UI element will also get the depth information of the nearest neighbor, so it will be affected by the following RawImage, etc.

It is assumed that the local server is up and running.

Please refer to the following for StartServer before use.

🖥️About local server

Set the base 3D models in the range of the camera

A parameter summary is shown below. Parameters that are adjusted frequently are bolded.

Parameter	Summary
Prompt	Enter the words you want to generate in natural language
NegativePrompt	Input elements in natural language that you do not want to include in the generated image
Size	Width and hight of the generated image in pixels
Steps	Increase this parameter if the generated image looks blurry
Cfg Scale	Strength to follow prompts Input larger value to strongly follow the prompt.
Seed	Fixing the seed value to generate a similar image In case of -1, the image will be random
Batch Count	The number of images to be generated. Select 1 to 100 images. Generated images will be stored in StreamingAssets/StableDiffusion. Please check images by date sort in the explorer.
Camera	Set the camera with setting up for getting depth
ControlType	You can choose Depth or Normal
Weight	You can adjust the intensity of ControlNet's influence.
Sampler	Changing the nuances of the generated image
Model	Base trained model See the "Switch Models/LoRA" to switch new model
Lora	Additional trained Lora to generate consitent specific pattern image See the "Switch Models/LoRA" to switch new LoRA If you select Lora, <lora:filename:1>will be added to the Prompt and LoRA will be available

Parameter

Summary

Prompt

Enter the words you want to generate in natural language

NegativePrompt

Input elements in natural language that you do not want to include in the generated image

Size

Width and hight of the generated image in pixels

Steps

Increase this parameter if the generated image looks blurry

Cfg Scale

Strength to follow prompts Input larger value to strongly follow the prompt.

Seed

Fixing the seed value to generate a similar image

In case of -1, the image will be random

Batch Count

The number of images to be generated.

Select 1 to 100 images.

Generated images will be stored in StreamingAssets/StableDiffusion. Please check images by date sort in the explorer.

Camera

Set the camera with setting up for getting depth

ControlType

You can choose Depth or Normal

Weight

You can adjust the intensity of ControlNet's influence.

Sampler

Changing the nuances of the generated image

Model

Base trained model See the "Switch Models/LoRA" to switch new model

Lora

Additional trained Lora to generate consitent specific pattern image

See the "Switch Models/LoRA" to switch new LoRA

If you select Lora, <lora:filename:1>will be added to the Prompt and LoRA will be available

PreviousHow to use Projector feature NextText to Image

Last updated 1 year ago