A service application includes a website host already setup for you. A service library is a library of services that a host can reference and startup.
If you start with a service library (recommended) you can then choose any host you wish (a windows service, IIS/ASP.NET, or even a console application) and you'd just reference your library from your new host. Choosing a Service Application limits your host to just IIS/ASP.NET (though this might be ok for your purposes, but will limit the protocols you can use).
Edit: Changes in IIS since I wrote this allow for a wider variety of protocols on ASP.NET activated services, so choosing a service application is much less limiting than before.