Scripting GDB to execute commands at particular breakpoints
This might be old news for the more experienced programmers out there, but yes, we can script GDB to do $stuff whenever it hits a breakpoint. With GDB's logging to file feature this can be super handy when trying to get a backlog of backtraces whenever a certain event arises.
Example use-case
Let's consider the following problem we'd like to debug: In KDevelop (Frameworks branch) we always got this annoying warning from Qt when exiting the application:
Output: QMutex: destroying locked mutex
Now, we can easily find out by grepping the Qt code base that this message is printed in qmutex.cpp:201 (which is inside ~QMutex). So, in order to figure out who's calling the destructor of QMutex and causing this output, let's put a breakpoint on qmutex.cpp:201 and re-run KDevelop and try to close it.
(gdb) break qmutex.cpp:201
Breakpoint 1 at 0x7ffff58f04bf: file /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp, line 201.
This leads to the following backtrace:
Breakpoint 1, QMutex::~QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, __in_chrg=) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:201
201 qWarning("QMutex: destroying locked mutex");
#0 QMutex::~QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, __in_chrg=) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:201
#1 0x00007ffff51638aa in __cxa_finalize (d=0x7ffff3428b78) at cxa_finalize.c:56
#2 0x00007ffff33f1573 in __do_global_dtors_aux () from /home/krf/devel/install/kf5/lib/x86_64-linux-gnu/libKDevPlatformUtil.so.9
#3 0x00007fffffffd830 in ?? ()
#4 0x00007ffff7dea73a in _dl_fini () at dl-fini.c:252
Unfortunately, the QMutex is destroyed during static deinitialization (notice the __do_global_dtors_aux call in the backtrace). Now, due to backtrace, we still don't know which QMutex in our code base got destroyed while being locked. We see that it is being statically initialized and must come out of libKDevPlatformUtil.so, but nothing more.
Problem: How do we find out which QMutex this was? Well, we need to check where this particular QMutex was first constructed.
GDB scripting to the rescue
I'd now like to print a backtrace each time we encounter the QMutex constructor (thus, QMutex::QMutex)
(gdb) break QMutex::QMutex
Breakpoint 2 at 0x7ffff58f040e: file /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp, line 178.
Additionally, I want to print a backtrace each time the breakpoint is encountered:
(gdb) command 2
Type commands for breakpoint(s) 2, one per line.
End with a line saying just "end".
>backtrace 10
>continue
>end
The command function makes GDB do the following each time it hits breakpoint 2: Print a backtrace limited to 10 frames and continue. (You can put whatever you need inside the command/end block.)
Furthermore, I'd like to get this logged to a file:
(gdb) set logging file gdb.txt
(gdb) set logging on
Copying output to gdb.txt.
(gdb) set pagination off
Now, let's restart KDevelop and close it again
(gdb) run
We'll again hit the breakpoint when printing the QMutex warning when static deinitialization happens:
Breakpoint 1, QMutex::~QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, __in_chrg=) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:201
201 qWarning("QMutex: destroying locked mutex");
#0 QMutex::~QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, __in_chrg=) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:201
#1 0x00007ffff51638aa in __cxa_finalize (d=0x7ffff3428b78) at cxa_finalize.c:56
#2 0x00007ffff33f1573 in __do_global_dtors_aux () from /home/krf/devel/install/kf5/lib/x86_64-linux-gnu/libKDevPlatformUtil.so.9
#3 0x00007fffffffd830 in ?? ()
#4 0x00007ffff7dea73a in _dl_fini () at dl-fini.c:252
Duly note the this pointer of the QMutex destroyed from the backtrace (QMutex::~QMutex (this=0x7ffff3428ba0 ...): It's 0x7ffff3428ba0
Note that in gdb.txt we now have the following contents (some parts replaced by ... for increased readability):
(...)
Breakpoint 2, QMutex::QMutex (this=0x7ffff7dd8b78 <(anonymous namespace)::resInit+24>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
178 QMutex::QMutex(RecursionMode mode)
#0 QMutex::QMutex (this=0x7ffff7dd8b78 <(anonymous namespace)::resInit+24>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
#1 0x00007ffff7be0e29 in (anonymous namespace)::ResInitUsage::ResInitUsage (this=0x7ffff7dd8b60 <(anonymous namespace)::resInit>) at /home/krf/devel/src/kf5/frameworks/kdelibs4support/src/kdecore/k3resolvermanager.cpp:166
#2 0x00007ffff7be2067 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/krf/devel/src/kf5/frameworks/kdelibs4support/src/kdecore/k3resolvermanager.cpp:237
#3 0x00007ffff7be2096 in _GLOBAL__sub_I_k3resolvermanager.cpp(void) () at /home/krf/devel/src/kf5/frameworks/kdelibs4support/src/kdecore/k3resolvermanager.cpp:815
#4 0x00007ffff7dea13a in call_init (...) at dl-init.c:78
#5 0x00007ffff7dea223 in call_init (...) at dl-init.c:36
#6 _dl_init (...) at dl-init.c:126
#7 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#8 0x0000000000000003 in ?? ()
#9 0x00007fffffffde39 in ?? ()
Breakpoint 2, QMutex::QMutex (this=0x7ffff7dd8b98 , mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
178 QMutex::QMutex(RecursionMode mode)
#0 QMutex::QMutex (this=0x7ffff7dd8b98 , mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
#1 0x00007ffff7be68fe in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/krf/devel/src/kf5/frameworks/kdelibs4support/src/kdecore/k3resolverstandardworkers.cpp:97
#2 0x00007ffff7be6956 in _GLOBAL__sub_I_k3resolverstandardworkers.cpp(void) () at /home/krf/devel/src/kf5/frameworks/kdelibs4support/src/kdecore/k3resolverstandardworkers.cpp:1049
#3 0x00007ffff7dea13a in call_init (...) at dl-init.c:78
#4 0x00007ffff7dea223 in call_init (...) at dl-init.c:36
#5 _dl_init (...) at dl-init.c:126
#6 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#7 0x0000000000000003 in ?? ()
#8 0x00007fffffffde39 in ?? ()
#9 0x00007fffffffde62 in ?? ()
Breakpoint 2, QMutex::QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
178 QMutex::QMutex(RecursionMode mode)
#0 QMutex::QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
#1 0x00007ffff33f23ba in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/krf/devel/src/kf5/extragear/kdevelop/kdevplatform/util/foregroundlock.cpp:29
#2 0x00007ffff33f24ab in _GLOBAL__sub_I_foregroundlock.cpp(void) () at /home/krf/devel/src/kf5/extragear/kdevelop/kdevplatform/util/foregroundlock.cpp:235
#3 0x00007ffff7dea13a in call_init (...) at dl-init.c:78
#4 0x00007ffff7dea223 in call_init (...) at dl-init.c:36
#5 _dl_init (...) at dl-init.c:126
#6 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#7 0x0000000000000003 in ?? ()
#8 0x00007fffffffde39 in ?? ()
#9 0x00007fffffffde62 in ?? ()
(...a lot more...)
Every time QMutex::QMutex was encountered, GDB printed a backtrace and logged it to the file.
Now, in order to find out where the QMutex comes from we simply search the string 0x7ffff3428ba0 inside gdb.txt and we'll find:
Breakpoint 2, QMutex::QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
178 QMutex::QMutex(RecursionMode mode)
#0 QMutex::QMutex (this=0x7ffff3428ba0 <(anonymous namespace)::internalMutex>, mode=QMutex::NonRecursive) at /home/krf/devel/src/qt5/qtbase/src/corelib/thread/qmutex.cpp:178
#1 0x00007ffff33f23ba in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /home/krf/devel/src/kf5/extragear/kdevelop/kdevplatform/util/foregroundlock.cpp:29
#2 0x00007ffff33f24ab in _GLOBAL__sub_I_foregroundlock.cpp(void) () at /home/krf/devel/src/kf5/extragear/kdevelop/kdevplatform/util/foregroundlock.cpp:235
#3 0x00007ffff7dea13a in call_init (...) at dl-init.c:78
#4 0x00007ffff7dea223 in call_init (...) at dl-init.c:36
#5 _dl_init (...) at dl-init.c:126
#6 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
Frame 2 shows: This mutex comes from /home/krf/devel/src/kf5/extragear/kdevelop/kdevplatform/util/foregroundlock.cpp:29, which says QMutex internalMutex;
We've found it!
At this point we can finally start solving our original problem of the destruction of a locked mutex, because now we at least know which mutex is causing this.
Other use-cases
Tracing ref-counting issues
You know that some object (for example QCoreApplication in Qt5) has a refcount higher than zero when exiting the application, but you don't know which object is still keeping a reference on it.
How to debug: Print backtraces each time we call the hypothetical ref() and deref() (for example QCoreApplication::{de}ref()). Now simply check which object never calls deref() in the GDB output file.
Verdict
GDB's scripting capabilities can be tremendously useful when attempting to debug issues where the backtrace at the point of crash or some other event just isn't enough.
This helped me to fix several issues in KDevelop already, which would have been hard to tackle otherwise.
Also see: https://sourceware.org/gdb/current/onlinedocs/gdb/Break-Commands.html