🎮Example of usage
Last updated
Last updated
Here is an example of how to use stable-diffusion-for-unity on the specific subject of 2D game background images.
Simple level design Modeling of Game System
Image generation by Depth2Img
Background image creation by Text2Img if necessary
Image editing + adjustment of generated images by Img2Img
We will generate image by Depth2Img using Depth obtained from the camera information in Unity.
So we need to set the camera and the Canvas for this.
Resolution for Game View At this time, we assume a side-scrolling game so set 1920x540 to generate a large width.
Camera settings
Set camera for depth. Remove the UI check on Rendering > Culling Musk so that the RawImage does not affect the Depth.
Change the projection to Orthographic
Adjust ClippingPlanes Near/Far Set the difference between Near<->Far to a smaller value. This will make it easier to reflect the Depth.
Create Canvas and RawImage
Chnage the Canvas's RenderMode to WorldSpace
Adjust the size of the Canvas and RawImage to fit the camera
Attach Text2ImageWithControlNet.cs to RawImage and set the size for 1920x540
Create the base shape of the stage for the game. At this time, we are using ProBuilder for modeling.
Note: ControlNet requires a large amount of VRAM. If you cannot generate images due to lack of VRAM, you may need to reduce the resolution.
Generate images according to the shape of the created 3D model.
Set the parameters in Text2ImageWithControlNet.cs and generate the image from Generate.
By selecting Depth/Normal in ControlType, it is possible to use the information from the camera position in Unity to generate an image that conforms to the 3D model shape in Unity.
Depth and Normal each generate images in accordance with the shape of the 3D model. Normal generates straight lines correctly, but parallel steps are less likely to be represented.
If you cannot generate images as expected, please adjust parameters such as CfgScale, Weight, etc., or set Model and LoRA. Click here to see how to set each parameter.
Please see this document for how to adjust these parameters
The recommended use is to apply LoRA that has learned the pattern to make it consistent across multiple stages. Note: The use of unauthorized images of others for LoRA and other model learning is an area where regulations have not yet caught up. Wictchpot Inc will not have responsibility for such actions. Please be respectful to other creators and use images with care so as not to infringe on others' copyrights.
Once a somewhat good image is generated, it can be adjusted by combining it with other tools like photoshop or Image to Image.
Make minor adjustments with combined with Img2Img or other image editing tool like photoshop.
Generated image is in StreamingAssets, so roughly modify this image in Photoshop.
Adjust this image using Image2Image or Inpaint. Since stable-diffusion-for-unity runs StableDiffusionWebUI, you can use the WebUI functions from the URL listed in the command prompt.
In this case, we used Image2Image and Inpaint to finalize the image.
After finalize
Input image is this
Final image is this
This image can be used as a background image for the game stage by placing characters and other objects on the original stage base model and setting SortingLayer and Order in layer appropriately.
| Input | Depth | Normal |
|---|---|---|
