In minimized form, Dojo is 136kb, jQuery is 96kb. Moving to dojo is not going in the right direction.
The issues you should consider are:
- Suitability of library for your purpose
- Size of library
- Likelihood that it will be precached already
- Your familiarity with the functionality of the library
- Availability on a popular, public CDN
- Good support on the net and great documentation
- Good reputation for reliability, cross browser support and regular updates
Go through each of these and unless you find another library that scores well on these, jQuery may be your best bet. It is surprisingly compact for what it offers you and it doesn't really have a lot of stuff that isn't in your list of things you want. jQuery has done a pretty good job of keeping the core library focused on its central mission and let UI stuff go into jQueryUI and most everything else into their own plug-in libraries.
If you're obsessed about optimizing the code you include to only be the things you need, then you may want to look at YUI. It was designed to be modular so that you can specify only the modules you want and then you can prebuild a chunk of code that only has those modules in it (or you can dynamically load just the modules you want). My sense is that YUI is somewhat overdesigned in this regard and it's cumbersome to use for quick projects because you have to spend the time to figure out which modules you need and generate that build each time. Once you get a bunch of modules loaded, it's not that compact either which is where you find that jQuery is surprisingly compact for what it includes.
In general, you should not worry about the things that a library includes that you are not using. Just look at the overall size and suitability of the libraries that do meet your needs. You can probably find a library that does only what you want and is bigger than jQuery and isn't widely cached so that wouldn't be a win.
There are compact libraries out there for just ajax or just deferred, but you probably want one with ajax and deferred implemented together so you can use deferred with ajax (like jQuery has done). Libraries that do extensive DOM manipulation tend to be more than just that because they are more designed to be your core library and most people have other needs besides just DOM manipulation.
In the end, I'd suggest that you shouldn't care what your library has in it that you don't need. Just evaluate it's overall suitability vs. the alternatives.