Modifying a Model to work with ARBodyTrackingConfiguration

Question

I have been working to learn how to perform 3D body tracking, using Apple's sample project. My goal is to replace the model in this project with my own 3D model, which I have built to match Apple's available robot.usdz sample included with that project.

I have successfully been able to import my 3D model into Xcode, after converting to USDZ, and attach the model to an ARBodyAnchor. However, my model (see attached screenshot) shows up in the expected location, but looks completely awry. I am loading the model using the following code:

var cancellable: AnyCancellable? = nil

cancellable = Entity.loadBodyTrackedAsync(named: "character/mytest").sink(
    receiveCompletion: { completion in
        if case let .failure(error) = completion {
            print("Unable to load model: \(error.localizedDescription)")
        }
        cancellable?.cancel()

        }, receiveValue: { (character: Entity) in
            if let character = character as? BodyTrackedEntity {
                character.scale = [0.5, 0.5, 0.5]
                self.character = character
                cancellable?.cancel()
            } else {
                print("Error: Unable to load model as BodyTrackedEntity")
            }
        })

When I import this same model simply as an Entity, taking away the body tracking, I can attach it to either a horizontal plane or a body, and it loads normally. Something, therefore, is awry with the model for the purpose of 3D tracking.

Has anyone had success loading a custom 3D model into Xcode for use with body tracking?

(Improper appearance of model when using body tracking) (Proper appearance of model when adding solely as an Entity - body tracking not enabled here)

Kono

Answer 1

Prepare a custom 3D character for AR Body Tracking

To use a custom character model (instead of Apple's robot.usdz) in real-time ARKit/RealityKit mocap, fulfil the following steps (and read carefully Rigging a model for MoCap article):

• Download robot biped model from Apple developer resource
• Open FBX robot version in Autodesk Maya 2024
• Unbind mesh from skeleton, then get rid of the robot mesh
• Leave all the skeleton's native joints' names and joints' hierarchy as they are
• Import your custom mesh into Maya
• Bind your custom mesh to the skeleton
• Using brush tool, paint new weights for each joint of your rigged model
• Export FBX character (mesh along with skeletal system)
• Convert your custom rigged character into USDZ model using Reality Converter
• Delight!

Here's the code:

import SwiftUI
import RealityKit
import ARKit

struct ContentView : View {
    var body: some View {
        ARContainer().ignoresSafeArea()
    }
}

---

struct ARContainer : UIViewRepresentable {
    var arView = ARView(frame: .zero)
    
    func makeUIView(context: Context) -> ARView {
        arView.automaticallyConfigureSession = false
        let config = ARBodyTrackingConfiguration()
        arView.session.run(config, options: .resetTracking)
        
        do {
            let body = try Entity.loadBodyTracked(named: "Custom_Model")
            let material_00 = SimpleMaterial(color: .white, isMetallic: true)
            let material_01 = OcclusionMaterial()
            body.model?.materials = [material_00, material_01]
            
            let anchor = AnchorEntity(.body)
            body.setParent(anchor)
            arView.scene.anchors.append(anchor)
        } catch {
            print("Can't load Body Tracked Entity...")
        }
        return arView
    }
    
    func updateUIView(_ view: ARView, context: Context) { }
}

USDZ model has two shaders: OcclusionMaterial for body, and SimpleMaterial for collar:

---

P. S.

The only issue is, the ARKit/RealityKit's realtime MoCap is very unstable (model often shakes uncontrollably) even when a visual system clearly distinguishes the tracked real-world body.

Stagemanage