These two libraries solve two different problems.
Bolts
Bolts simplifies asynchronous programming by transparently pushing code to a background thread. Bolts also spends a good deal of effort attempting to reduce unsightly code nesting that produces a nested pyramid like format.
Therefore, if you are specifically looking to deal with async (multi-threading) issues, Bolts is on of the more robust solutions. Bolts is also effective at relieving code nesting and callback boilerplate and is probably a great solution for just trying to relieve callback issues.
RxJava
RxJava is specifically designed to support a reactive programming paradigm. Reactive programming is an alternative to imperative programming in Java. You might choose to move to a reactive programming paradigm for various reasons - of which there are many. If you want migrate your code to the reactive programming model, or you want to use reactive in your greenfield projects, consider using RxJava - the de facto reactive standard in the Java world.
Reactive does also solve the asynchronous programming problem, as well reduce callback boilerplate via generics. But it should not be used just to solve those problems. For example, Bolts ability to solve the nested pyramid code structure makes it a more viable solution for async programming. On the other hand, if you are using reactive via RxJava, async problems are already solved, so there is no point bringing in Bolts.