The short answer is I wouldn't bother trying! This is because Alloy UI is so integral to the Liferay UI, without it you'd have a lot of work to do to replace with jQuery replacements.
Alloy UI does a lot more than just draggable portlets, and sorting. It also controls popups, AJAX calls, and loads of other stuff that I haven't come across yet.
HOWEVER if you really want to put yourself through it then I'd suggest you first of all create a Custom Theme and remove any references to Alloy UI from the theme, and load jQuery instead.
Then you'll also need to create a hook or hooks (and possibly use EXT plugins) to replace all of the JSP files that reference Alloy UI, with copies that use jQuery to do the same functionality.
OR another way would be to for the Liferay source code on GitHub and create your own version from the source removing Alloy UI.
BUT I still strongly suggest you don't bother wasting your time. Love it or hate it, Alloy UI is bake dried into Liferay. If you follow either of my suggestions you're effectively stopping yourself from upgrading Liferay in the future.
A better suggestion would be to suggest ways of improving Alloy UI to Liferay, or contribute to the Alloy UI repository on GitHub and help the community.
~~ EDIT ~~
To disable Alloy UI for just non-authenticated users, then perhaps a HOOK that modifed the files in portal-web\docroot\html\common\themes\ would do the trick. In particular look at top_js.jspf. If you put checks in there to say only load Alloy UI if you're signed in then it might be the answer your after.