I experienced a similar issue but the problem, in my case, is not related to speed but something different.
Specifically, if you force your iOS Simulator to run under Rosetta you'll see that it works as expected. Not sure why, but this is what I'm experiencing. Always reproducible with 1.22.5 (stable).
To run in Rosetta, right click on Xcode and choose "Show Package Contents", from there you navigate to "Contents > Developer > Applications," There you'll find the Simulator app. If you right click on it and choose "Get Info", you'll find an option to run it using Rosetta.
Update 14/9/2021: Xcode 12.5.1 no longer needs the above workaround. Scrolling is working fine out-of-the-box.
Update 29/10/2023: Xcode 15.x and possible other versions in the past, unfortunately seems to suffer again of the infamous bug above.
I found that running Simulator with x86_64 emulation on Ventura, Sonoma as follow:
arch -x86_64 /Applications/Xcode.app/Contents/Developer/Applications/Simulator.app/Contents/MacOS/Simulator
workaround the problem.