C++ thread not stopping in gdb async mode using user-defined or python command sequence

Question

I'm using gdb 7.4.1 on embedded powerpc target to perform some analysis on my multi-threaded C++ program that uses pthreads. My end goal is to script gdb with python to automate some common analysis functions. The problem is that I am finding some discrepancy in behavior when I run commands individually vs. in a gdb user-defined command (or invoking the same commands via python script).

edit: I found this reference to a very similar problem on the main gdb mailing list. Although I don't completely follow Pedro's response about the limitation of async mode, I think he's implying that in async mode, the relative timing of user-defined command sequences cannot be trusted. This is what I found empirically.

In both scenarios, I perform the following start-up steps, loading my program, setting its args, and turning on asynchronous and non-stop debugging modes, then running the program in the background:

(gdb) file myprogram
(gdb) set args --interface=eth0 --try-count=0
(gdb) set target-async on
(gdb) set pagination off
(gdb) set non-stop on
(gdb) run &

At this point, if I manually issue interrupt and then info threads commands, I see the list of all threads running except one that got stopped. Then I can continue & and repeat to my hearts content, it works consistently. When stopped, I can inspect that thread's stack frames and all is well.

However, if instead I put these commands into a user-defined gdb command:

(gdb) define foo
(gdb) interrupt
(gdb) info threads
(gdb) continue &
(gdb) end
(gdb) foo
Cannot execute this command while the selected thread is running.

Then the thread list printed by foo indicates no threads were stopped, and so the continue & command returns Cannot execute this command while the selected thread is running.. I thought this was a problem inherent to the asynchronous gdb commanding, so I inserted an absurdly long wait after the interrupt command and got the same behavior:

(gdb) define foo
(gdb) interrupt
(gdb) shell sleep 5
(gdb) info threads
(gdb) continue &
(gdb) end
(gdb) foo
Cannot execute this command while the selected thread is running.

With or without the sleep command, I can always issue the manual CLI commands and the threads get stopped correctly.

Similarly, I get the same results sourcing a python script to do the thread perusal:

import gdb, time

gdb.execute("file myprogram")
gdb.execute("set args --interface=eth0 --try-count=0")
gdb.execute("set target-async on")
gdb.execute("set pagination off") 
gdb.execute("set non-stop on")
gdb.execute("run &")
time.sleep(5)
gdb.execute("interrupt")

# here, I inspect threads via gdb module interface
# in practice, they're always all running bc the program neven got interrupted
for thread in gdb.selected_inferior().threads():
    print thread.is_running(),

gdb.execute("continue &")

I get the same result even if I specify from_tty=True in the gdb.execute calls. Also, if I use continue -a it suppresses the error string but does not help otherwise bc the interrupt call still doesn't work.

So... is this:

• cockpit error? Is there something that I'm omitting or doing incorrectly, given what I'm trying to accomplish? Should this work, or do I have to use GDB/MI to asynchronously "drive" gdb like this?
• a timing problem? Maybe invoking shell sleep (or python time.sleep()) doesn't do what I assume it would, in this context.
• problem with my usage of pthreads? I have assumed that since using manual gdb commands always works correctly this is not the case.
• a gdb problem?

Thanks.

Castaneda

Answer 1

I think this is most likely a gdb problem. I don't know enough about the inferior-control stuff to be more confident. I do know that inferior control generally has not been wired up to Python...

One thing worth trying is having a separate Python thread that does the wait, then sends an "interrupt" command to the main gdb thread using gdb.post_event.

Then, instead of synchronously examining the threads or doing work after the "interrupt", instead use the gdb.events.stop event source to trigger your actions.

Please file bugs liberally about holes in the Python API.

Nolasco

Answer 2

I came across your post today and tested this gdb-python combination. I found a way to make it works.

1. Set an env in your terminal etc: setenv MY_ENV 1
2. In your C/C++ source code, you can add a few line near the start of the code to get the env: getenv(MY_ENV), and then print out the process id with get_pid()
3. Set the env in the terminal, then run the program. The program will stop and show its process id on the screen. say 1234
4. Write a separate gdb_monitor.py for gdb e.g:

 import gdb
 # Your code
 gdb.execute("set target-async on")
 gdb.execute("set pagination off") 
 gdb.execute("set non-stop on")
 gdb.execute("set args --interface=eth0 --try-count=0")
 # Add these lines
 pid = input("Python script is running, attach the pid here:")
 gdb.execute("attach {}".format(pid))

while(1):
 print("I am running gdb in async mode, add the break condition here")

5. Now in gdb, you can source the python script, enter the pid in the script.

Amphibious