In terms of drawbacks, The primary disadvantage of serialization is the performance overhead (both CPU and the disk) and the potential latency issues when sending it over the wire. There may be slight concerns with security because in general, XML serialization is insecure since it works only on public properties and classes, forcing you in some cases to exposed properties you may not have otherwise. Of course if security is really a concern, you probably wouldn't be storing too sensitive of data in session.
If you are using Silverlight, one potential drawback is that Silverlight does not support the [Serializable] attribute, so any classes decorated with it would be unusable for your Silverlight assemblies.
That said, for session management, small objects stored in the ASPState database typically perform just fine without any noticeable difference over in memory session. On the opposite end of the spectrum, I have had large objects with lists of other objects as properties etc, and if they are big enough, the performance hit can be noticeable at times.
[Serializable]
– Branum