How to use multiple animations on a USDZ model with RealityKit?

Question

How to use multiple animations for a USDZ 3D character in RealityKit?

Play an idle animation, when it walks, when it jumps, etc.

Materials

Answer 1

Switch between Multiple Animations in RealityKit

In RealityKit 2023, you can switch between multiple animations like idle, dance or jump, but for that you need the same skeletal structure in each USDZ file. To implement it, you have to put all the animations from several animated USDZ models into AnimationResource collection, and then apply one animation at a time to a character with an identical skeleton. Blending works perfectly here.

You can test my code with Mixamo animations, where each character has an identical skeleton.

I downloaded several animated Mixamo models in .dae format two-three years ago, and after that made a DAE -> SCN -> USDZ conversion using a code published in this post.

import SwiftUI
import RealityKit

struct ContentView : View {

    @State var animationResource: [AnimationResource] = []
    @State var model1 = try! Entity.loadModel(named: "idle.usdz")
    @State var model2 = try! Entity.loadModel(named: "jump.usdz")
    
    var body : some View {
        ZStack {
            ARViewContainer(model: $model1)
                .ignoresSafeArea()
                .onAppear {
                    animationResource.append(model1.availableAnimations[0])
                    animationResource.append(model2.availableAnimations[0])
                }
            VStack {
                Spacer()
                HStack {
                    Spacer()
                    Button("Jump") {
                        model1.playAnimation(animationResource[1].repeat(),
                                             transitionDuration: 0.5)
                    }
                    Spacer()
                    Button("Neutral") {
                        model1.playAnimation(animationResource[0].repeat(),
                                             transitionDuration: 0.5)
                    }
                    Spacer()
                }
            }
        }
    }
}

---

struct ARViewContainer : UIViewRepresentable {

    @Binding var model: ModelEntity
    let arView = ARView(frame: .zero)
    let anchor = AnchorEntity()

    func makeUIView(context: Context) -> ARView {
        model.scale /= 120
        model.position.y = -0.7
        anchor.addChild(model)
        arView.scene.anchors.append(anchor)
        return arView
    }
    func updateUIView(_ view: ARView, context: Context) { }
}

#Preview {
    ContentView()
}

Pineal

I don’t understand… It doesn't work for me… This is my playground – Materials

@jeremyjoron, your Salsa.usdz file is damaged or corrupted. Or its skeleton is different from Idle.usdz file. Try another models – I published a link to Mixamo. – Pineal

I downloaded a new model directly from mixamo in fbx with 3 animations. I converted to usdz with Reality Converter and I have exactly the same problem. In addition, when I try the same manipulation with SceneKit the animations work well so I don't think it's a problem with the files… – Materials

According to messages in Xcode's console, we have a problem with a model – Cannot find a BindPoint for any bind path: "", "", "Salsa_Dancing.Agnes.mixamorig_Hips", "Salsa_Dancing.Agnes.mixamorig_Hips.SkeletalPose.SkeletalPoses[0]". I agree that RealityKit is a more capricious SDK than SceneKit in animation sense. – Pineal

Oh thank you ! I use Swift Playground on iPad i didn’t view this error. I will to try without mixamo… – Materials

2024-Export FBX from Mixamo, converting by Reality Converter to USDZ, importing to Reality Composer Pro, retargeting animation from one to another, finally export the Scene to USDZ twice（once a animation）,and using to code up here！I done！ – Needlewoman

Answer 2

My intention is to present multiple animations in the app, so I place the "idle" and "walk" actions in order on the Blender timeline, export them to USDC, and convert them to USDZ through Reality Converter. Then, I use the following code to extract a single AnimationResource from the timeline.

        .onAppear {
            var animationResource: [AnimationResource] = []
            var model3 = try! Entity.loadModel(named: "v3.usdz")
            let definition = model3.availableAnimations[0].definition
            
            //🧍
            let idle = AnimationView(source: definition,
                name: "idle",
                bindTarget: nil,
                blendLayer: 0,
                repeatMode: .autoReverse,
                fillMode: [],
                                     trimStart: 0.0,
                trimEnd: 2.0,
                trimDuration: nil,
                offset: 0,
                delay: 0,
                speed: 1.0)
            
            //🚶
            let walk = AnimationView(source: definition,
                name: "walk",
                bindTarget: nil,
                blendLayer: 0,
                repeatMode: .autoReverse,
                fillMode: [],
                                     trimStart: 2.0,
                                     trimEnd: 10.0,
                trimDuration: nil,
                offset: 0,
                delay: 0,
                speed: 1.0)


            // Create an animation resource from the clip.
            var clipResource = try! AnimationResource.generate(with: idle)
            animationResource.append(clipResource)
        }

Melanite

Alert！This method will break when making transition between animation. – Needlewoman