It's been a while since I've delved into that sort of stuff, but assuming you are correct that the following address is pushed on the stack, the debugger can pop the return address and use it to figure out where the breakpoint was (the return address minus one, since the INT 3 instruction is one byte long) [edited].
In other words, the debugger doesn't necessarily need to return to the address on the stack. It can restore the original instruction, and then execute it at the original location. If the breakpoint is to remain set, it can use the "trap bit" in the flags to execute only a single instruction - the original one that was overwritten - before another trap will be generated (INT 3 again I think); then the INT 3 instruction can be re-established before continuing execution properly.
Most of the time, though, debuggers are operating under a system where they're not directly handling the trap anyway; they might be delivered a signal, for instance, telling them where the trap occurred. Most likely they still need to figure out the "real" address (i.e. the address of the INT 3 instruction) from the trap address, as the OS has no way to do this.
Things get complicated, too, if there are multiple threads involved; in that case restoring the original instruction "in place" may lead to the breakpoint being missed if it is hit by another thread. One solution might be stopping all the other threads before restoring the instruction (and starting them again afterwards).