On OS X I had a similar sounding problem where flymake would hang emacs when opening a new file (even with only a few buffers open). Enabling debug-on-quit
didn't help, as the entire GUI locked up.
Attaching gdb to emacs returned the following:
$ gdb -p `psgrep emacs`
(gdb) bt
#0 0x00007fff98954e42 in __semwait_signal ()
#1 0x00007fff8e5d1dea in nanosleep ()
#2 0x00007fff99e3af05 in +[NSThread sleepUntilDate:] ()
#3 0x000000010015d917 in -[EmacsDialogPanel runDialogAt:] ()
#4 0x000000010015fa1f in ns_popup_dialog ()
[...]
So the lockup was caused by a dialog message saying "Configuration error has occurred while running ..." or similar
From this post describing the problem, you can disable these popup messages with the following:
Sometimes, however, Emacs just hangs. The OS X red/yellow/blue close/minimize/zoom buttons turn grey as if Emacs has lost focus, but it is in fact the active application. Switching tasks around and back to Emacs doesn't recover it. You can't Command-Q quit it.
When you do finally give up and kill it and restart, you've lost your work, your shells, and it doesn't save the list of recent-files.
(setq flymake-gui-warnings-enabled nil)
This prevents flymake from alerting you that it's switching itself off, but it's better than the alternative.
The flymake-display-warning calls Emac's 'message-box' function so there may be other tools which cause the same problem.
There's also a generic variable use-dialog-box but setting it from default t to nil didn't help with flymake. There's also a suggestion on the interwebs to use defadvice to tell y-or-no-p and yes-or-no-p set use-dialog-box to nil, but again, this won't help flymake since it doesn't use those functions.