# Text to Image With ControlNet

Generate images based on depth or normal information obtained from the Unity camera

| Input                                                                                                                                                                                                                 | Depth                                                                                                                                                                                                                                              | Normal                                                                                                                                                                                                                                             |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ![](https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2FeuUs7DOTjfp3l3RRMVc1%2F3Dmodel.png?alt=media\&token=11d1025e-1f72-4a97-b5a5-de902d400f77) | ![](https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2FEO2LzLIQPVLrDGU3teNV%2Ff9b2bf9f-791c-422c-b18f-edd4ea40b547.png?alt=media\&token=9de9de98-5e5e-4ccb-a37e-4873f7658bbb) | ![](https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2Ff6S4Zq1tvi6u4uRurStl%2Ff7b05689-170a-4570-b5c4-746659211569.png?alt=media\&token=193bea07-9880-4e13-95c0-3702dc889e84) |

This makes it possible to use image generation in areas related to the game system where geometry is important. 

Note: ControlNet requires a large amount of VRAM. If you are unable to generate images due to insufficient VRAM, you may need to reduce the resolution.<br>

<figure><img src="https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2F9lszuZuuQ2iLaQKilZd0%2FText2ImageWithControlNet.png?alt=media&#x26;token=ff6a3843-cc25-4798-baab-93c55fd8d039" alt=""><figcaption></figcaption></figure>

For generate image with Depth to Image, attach the Text2ImgWithControlNet.cs component to the object

Uncheck the CullingMask UI checkbox.&#x20;

If you do not do this, the UI element will also get the depth information of the nearest neighbor, so it will be affected by the following RawImage, etc.

<figure><img src="https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2F1m839XGRALatwrW0F8W4%2Fimage.png?alt=media&#x26;token=4a9a6637-f427-41f7-ad94-803c698ed3dc" alt=""><figcaption></figcaption></figure>

It is assumed that the local server is up and running.&#x20;

Please refer to the following for StartServer before use.

{% content-ref url="../production-tips/about-local-server" %}
[about-local-server](https://docs.witchpot.com/english/production-tips/about-local-server)
{% endcontent-ref %}

Set the base 3D models in the range of the camera

<figure><img src="https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2FXhEbuzoeEdT7Po7y6okC%2Fimage.png?alt=media&#x26;token=61b8fe99-78bf-4786-83c6-19fd874b2df8" alt=""><figcaption><p>camera range</p></figcaption></figure>

<figure><img src="https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2FR8ifKMWszKx6HeLfnscM%2Fimage.png?alt=media&#x26;token=18bbf665-b272-4acd-b9f5-467fa9b510f0" alt=""><figcaption><p>place 3D model in the range of camera</p></figcaption></figure>

<figure><img src="https://1270527639-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FtVdP5UEfu2PNFw4I6asq%2Fuploads%2FjiiU3YLRA0F06pUOTnUX%2Ffinalimage.png?alt=media&#x26;token=3bdc0326-4665-46af-8c7a-8bc645d2d013" alt=""><figcaption></figcaption></figure>

A parameter summary is shown below. Parameters that are adjusted frequently are bolded.

<table><thead><tr><th width="177">Parameter</th><th>Summary</th></tr></thead><tbody><tr><td><strong>Prompt</strong></td><td>Enter the words you want to generate in natural language</td></tr><tr><td><strong>NegativePrompt</strong></td><td>Input elements in natural language that you do not want to include in the generated image</td></tr><tr><td><strong>Size</strong></td><td>Width and hight of the generated image in pixels</td></tr><tr><td>Steps</td><td>Increase this parameter if the generated image looks blurry</td></tr><tr><td><strong>Cfg Scale</strong></td><td>Strength to follow prompts<br>Input larger value to strongly follow the prompt.</td></tr><tr><td>Seed</td><td><p>Fixing the seed value to generate a similar image</p><p>In case of -1, the image will be random</p></td></tr><tr><td><strong>Batch Count</strong></td><td><p>The number of images to be generated.</p><p>Select 1 to 100 images.</p><p>Generated images will be stored in StreamingAssets/StableDiffusion. Please check images by date sort in the explorer.</p></td></tr><tr><td><strong>Camera</strong></td><td>Set the camera with setting up for getting depth</td></tr><tr><td><strong>ControlType</strong></td><td>You can choose Depth or Normal</td></tr><tr><td><strong>Weight</strong></td><td>You can adjust the intensity of ControlNet's influence.</td></tr><tr><td>Sampler</td><td>Changing the nuances of the generated image</td></tr><tr><td><strong>Model</strong></td><td>Base trained model<br>See the "<a href="../production-tips/switch-models-lora">Switch Models/LoRA</a>" to switch new model</td></tr><tr><td><strong>Lora</strong></td><td><p>Additional trained Lora to generate consitent specific pattern image</p><p>See the "<a href="../production-tips/switch-models-lora">Switch Models/LoRA</a>" to switch new LoRA</p><p>If you select Lora, &#x3C;lora:filename:1>will be added to the Prompt and LoRA will be available</p></td></tr></tbody></table>