🎮Example of usage

Here is an example of how to use stable-diffusion-for-unity on the specific subject of 2D game background images.

Summary

• Simple level design Modeling of Game System

• Image generation by Depth2Img

• Background image creation by Text2Img if necessary

• Image editing + adjustment of generated images by Img2Img

Preparation

We will generate image by Depth2Img using Depth obtained from the camera information in Unity.

So we need to set the camera and the Canvas for this.

• Resolution for Game View At this time, we assume a side-scrolling game so set 1920x540 to generate a large width.

  

  
• Camera settings

  • Set camera for depth. Remove the UI check on Rendering > Culling Musk so that the RawImage does not affect the Depth.

    
  • Change the projection to Orthographic

    
  • Adjust ClippingPlanes Near/Far Set the difference between Near<->Far to a smaller value. This will make it easier to reflect the Depth.

    
• Create Canvas and RawImage

• Chnage the Canvas's RenderMode to WorldSpace

  
• Adjust the size of the Canvas and RawImage to fit the camera

  
• Attach Text2ImageWithControlNet.cs to RawImage and set the size for 1920x540

  

Simple level design Modeling of Game System

Create the base shape of the stage for the game. At this time, we are using ProBuilder for modeling.

Image Generation with Depth2Img

Note: ControlNet requires a large amount of VRAM. If you cannot generate images due to lack of VRAM, you may need to reduce the resolution.

Generate images according to the shape of the created 3D model.

Set the parameters in Text2ImageWithControlNet.cs and generate the image from Generate.

By selecting Depth/Normal in ControlType, it is possible to use the information from the camera position in Unity to generate an image that conforms to the 3D model shape in Unity.

Depth and Normal each generate images in accordance with the shape of the 3D model. Normal generates straight lines correctly, but parallel steps are less likely to be represented.

If you cannot generate images as expected, please adjust parameters such as CfgScale, Weight, etc., or set Model and LoRA. Click here to see how to set each parameter.

Please see this document for how to adjust these parameters

The recommended use is to apply LoRA that has learned the pattern to make it consistent across multiple stages. Note: The use of unauthorized images of others for LoRA and other model learning is an area where regulations have not yet caught up. Wictchpot Inc will not have responsibility for such actions. Please be respectful to other creators and use images with care so as not to infringe on others' copyrights.

Once a somewhat good image is generated, it can be adjusted by combining it with other tools like photoshop or Image to Image.

Final image adjustment

Make minor adjustments with combined with Img2Img or other image editing tool like photoshop.

Generated image is in StreamingAssets, so roughly modify this image in Photoshop.

Adjust this image using Image2Image or Inpaint. Since stable-diffusion-for-unity runs StableDiffusionWebUI, you can use the WebUI functions from the URL listed in the command prompt.

In this case, we used Image2Image and Inpaint to finalize the image.

After finalize

Input image is this

Final image is this

This image can be used as a background image for the game stage by placing characters and other objects on the original stage base model and setting SortingLayer and Order in layer appropriately.