How to Add Material to ModelEntity programatically in RealityKit?

Question

The docs for RealityKit include the structs: OcclusionMaterial, SimpleMaterial, and UnlitMaterial for adding materials to a ModelEntity.

Alternatively you can load in a model with a material attached to it.

I want to add a custom material/texture to a ModelEntity programmatically. How can I achieve this on the fly without adding the material to a model in Reality Composer or some other 3D Software?

Waiver

Answer 1

Updated: January 07, 2024

RealityKit materials

There are eight types of materials in RealityKit at the moment:

• SimpleMaterial
• UnlitMaterial
• OcclusionMaterial (read this post to find out how to setup SceneKit occlusion shader)
• VideoMaterial (look at this post to find out how to apply it)
• PhysicallyBasedMaterial
• CustomMaterial (read my Medium story)
• ShaderGraphMaterial (also known as MaterialX, works in visionOS only)
• PortalMaterial (works in visionOS only)

SwiftUI version

Here I used two macOS implementations (SwiftUI and Cocoa) to demonstrate how to programmatically assign RealityKit materials.

import SwiftUI
import RealityKit

struct VRContainer : NSViewRepresentable {        
    let arView = ARView(frame: .zero)
    let anchor = AnchorEntity()
    
    func makeNSView(context: Context) -> ARView {                    
        var smpl = SimpleMaterial()
        smpl.color.tint = .blue
        smpl.metallic = 0.7
        smpl.roughness = 0.2
                
        var pbr = PhysicallyBasedMaterial()
        pbr.baseColor.tint = .green

        let mesh: MeshResource = .generateBox(width: 0.5,
                                             height: 0.5,
                                              depth: 0.5,
                                       cornerRadius: 0.02,
                                         splitFaces: true)

        let box = ModelEntity(mesh: mesh, materials: [smpl, pbr])    
        box.orientation = Transform(pitch: .pi/4,
                                      yaw: .pi/4, roll: 0.0).rotation
        anchor.addChild(box)
        arView.scene.anchors.append(anchor)
        arView.environment.background = .color(.black)
        return arView
    }
    func updateNSView(_ view: ARView, context: Context) { }
}

struct ContentView: View {
    var body: some View {
        VRContainer().ignoresSafeArea()
    }
}

Cocoa version

import Cocoa
import RealityKit

class ViewController: NSViewController {        
    @IBOutlet var arView: ARView!
    
    override func awakeFromNib() {
        let box = try! Experience.loadBox()
        
        var simpleMat = SimpleMaterial()
        simpleMat.color = .init(tint: .blue, texture: nil)
        simpleMat.metallic = .init(floatLiteral: 0.7)
        simpleMat.roughness = .init(floatLiteral: 0.2)
        
        var pbr = PhysicallyBasedMaterial()
        pbr.baseColor = .init(tint: .green, texture: nil) 

        let mesh: MeshResource = .generateBox(width: 0.5, 
                                             height: 0.5, 
                                              depth: 0.5, 
                                       cornerRadius: 0.02, 
                                         splitFaces: true)

        let boxComponent = ModelComponent(mesh: mesh,
                                     materials: [simpleMat, pbr])

        box.steelBox?.children[0].components.set(boxComponent)
        box.steelBox?.orientation = Transform(pitch: .pi/4, 
                                                yaw: .pi/4, 
                                               roll: 0).rotation
        arView.scene.anchors.append(box)
    }
}

Read this post to find out how to load a texture for RealityKit's shaders.

RealityKit shaders vs SceneKit shaders

We know that in SceneKit there are 5 different shading models, so we can use RealityKit's SimpleMaterial, PhysicallyBasedMaterial and UnlitMaterial to generate all these five shaders that we've been accustomed to.

Let's see how it looks like:

SCNMaterial.LightingModel.blinn           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.5),
                                                       isMetallic: false)

SCNMaterial.LightingModel.lambert         – SimpleMaterial(color: . gray,
                                                        roughness: .float(1.0),
                                                       isMetallic: false)
  
SCNMaterial.LightingModel.phong           – SimpleMaterial(color: . gray,
                                                        roughness: .float(0.0),
                                                       isMetallic: false)

SCNMaterial.LightingModel.physicallyBased – PhysicallyBasedMaterial()


// all three shaders (`.constant`, `UnlitMaterial` and `VideoMaterial `) 
// don't depend on lighting
SCNMaterial.LightingModel.constant        – UnlitMaterial(color: .gray)
                                          – VideoMaterial(avPlayer: avPlayer)

Regress

I can confirm this works perfectly in iOS 13! Thanks @ARGeo – Waiver

It does work, yes. Making the alpha channel work seems to rely on a workaround as of now: Apple forums – Amazing

Is there any way to add 6x face textures to a box? – Judah

Yeah, I’m not sure how to use 3D software. I was hoping there was a easy way to do it in code. Like SceneKit can take an array of 6 textures to put on each face of a box. – Judah

Hi @JaydenIrwin, I've updated my answer. There's a splitFaces: Bool parameter that could help you apply 6X texture. – Regress

@AndyJazz I have tried to put different materials (VideoMaterial and SimpleMaterial) on a split faced box, however, this shows cubes sides with video textures only (others are invisible). If I don't mix the materials the result is fine. I've done it on VisionOS simulator. Maybe this is the problem? – Bratislava