I think this is a good question, and it's something I'm trying to work out for my own research as well.
There are, at the moment, strong limitations in terms of what GPUs can do, as they require individual threads to execute exactly the same code on different sets of data, i.e. the problem / algorithm must be "data parallel". Obviously data parallel problems include Monte Carlo simulations (where many MC simulations are executed in parallel), image processing, and less obviously molecular dynamics simulations. Numerical integration (Monte Carlo or otherwise) is another scientific application which can be easily ported to running on a GPU.
The other main restriction is that memory per thread is very limited, and so to be efficiently executed on a GPU the algorithm must have high arithmetic intensity. A necessary but not sufficient condition for an algorithm to be a candidate for running on a GPU is that on the CPU the algorithm must be strongly CPU bound rather than memory bound.
My view is that as time goes on, more and more problems will be shoehorned so that they can be solved using this paradigm just because there is such a large performance gain to be made, but the low hanging fruit are the obviously data parallel problems. Massively multicore programming is, in my view, going to be increasingly important and prevalent in scientific circles over the next decade.
I've played around with this a bit, and managed to shoehorn a backtracking problem into an appropriate format for executing on a GPU (using CUDA). FYI, I describe this in a talk: http://lattice.complex.unimelb.edu.au/home/sites/default/files/mydocuments/clisby_cuda0509.pdf
