As I describe here, I've used both UIViews and CALayers in animations and found a negligible performance difference between them. UIViews are very lightweight wrappers around the layers. Also, any layer-based animations you need can be applied to a UIView's backing layer easily.
I've used CALayers directly in situations where I wanted to create cross-platform (Mac / iOS) UI elements, because CALayers are almost identical in their implementation on both platforms (unlike the significantly different NSViews and UIViews). CALayers don't have any touch-handling routines out of the box, but you can add that capability if you need to.
There are also some edge cases where you might want to work directly with layers, like when trying to do limited 3-D manipulation of the layers (as in a CoverFlow effect) or when using a CAReplicatorLayer to produce particle effects.