GDB Tips and Tricks #4: Reverse Debugging

How many times have you stepped through code only to find that you’ve gone too far? Maybe you overstepped a critical point in the code’s execution. Or perhaps you stepped over a function you intended to step into.

Did you know that GDB could actually step backwards through of code? That’s right. You can have GDB go back in time. This is often referred to as “reverse debugging.” But how does it work?

How It Works

Reverse debugging actually relies upon another gem in GDB’s bag-of-tricks called “process record and replay”. Rolls right off the tongue doesn’t it? I won’t spend a lot of time going into the details of PRR here, but it’s quite powerful. The only PRR command we need to be concerned with in this discussion is “record”.

The “record” command begins recording the execution of your application, making notes of things like memory and register values. When you arrive at a point in your application at which you’d like to go backwards, you can issue the “reverse” versions of all the navigation commands you’re already familiar with. This include reverse-step (rs), reverse-next (rn), reverse-continue (rc), and reverse-finish (no short version 🙁 ). As you move backwards through code, gdb reverts the state of memory and registers, effectively unexecuting lines of code.

Let’s see an example using the code snippet below.

#include <iostream>
int sum(int a, int b)
    int result = a + b;
    return result;
int main(int argc, char **argv)
    int a = 12;
    int b = 13;
    int c = sum(a, b);
    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";
    return 0;

Compile this (don’t forget to compile it with the ‘-g’ flag!) and fire up gdb. Then set a breakpoint at main. We can’t begin recording program execution before it’s actually running. So we issue the run command, which will execute our application and promptly break at main.

(gdb) break main
Breakpoint 1 at 0x4007df: file gdbtest.cpp, line 11.
(gdb) run
Starting program: /home/skirk/gdbtest 
Breakpoint 1, main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:11
11	    int a = 12;

At this point, we issue the “record” command to begin recording.

(gdb) record

Now let’s start stepping through the code.

(gdb) n
12	    int b = 13;
(gdb) n
13	    int c = sum(a, b);
(gdb) n
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

We’re now at the point just before the sum is written to stdout. What if I had intended to step into the sum function to see what it’s doing? Let’s back up to just before the sum function is called and then step into it.

(gdb) reverse-next
13	    int c = sum(a, b);
(gdb) s
sum (a=12, b=13) at gdbtest.cpp:5
5	    int result = a + b;

Now we appear to have gone back into time. This allows us to step into the sum function. At this point, we can inspect the values of parameters a and b as we normally would.

(gdb) print a
$1 = 12
(gdb) print b
$2 = 13

If we’re satisfied with the state of things, we can allow the program to continue on.

(gdb) c
No more reverse-execution history.
main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:15
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

An interesting thing happened here. The program execution stopped at the point at which we previously started stepping backwards. When stepping through code using recorded history, “continue” will continue program execution until the history has been exhausted, unless, of course, it has some other reason to stop such as breakpoints and the like.

Let’s now stop the recording process using the “record stop” command and allow the program to continue execution until completion.

(gdb) record stop
Process record is stopped and all execution logs are deleted.
(gdb) c
The sum of 12 and 13 is 25
[Inferior 1 (process 10608) exited normally]

What if we hadn’t stopped recording? Well, it depends. If your version of the runtime executes instructions that aren’t supported by PRR, then you may encounter errors such as this…

Process record does not support instruction 0xc5 at address 0x7ffff7dee8b7.
Process record: failed to record execution log.
Program stopped.
_dl_runtime_resolve_avx () at ../sysdeps/x86_64/dl-trampoline.h:81
81	../sysdeps/x86_64/dl-trampoline.h: No such file or directory.

In this case, AVX instructions are being executed which aren’t supported by the record process. (In this particular case, there’s a workaround. We can export the environment variable LD_BIND_NOW=1 which resolves all symbols at load time. Doing so actually prevents the call to _dl_runtime_resolve_avx later.)

It’s also possible you might see something like…

The sum of 12 and 13 is 25
The next instruction is syscall exit_group.  It will make the program exit.  Do you want to stop the program?([y] or n)

Here you’re prompted as to whether or not you want to stop the program. Regardless of what you choose, you’re still able to navigate backwards in program execution. That’s right – you can reverse debug an application that has finished running.


There are a few caveats when performing reverse debugging.

The first is that you can’t move backwards beyond the point at which you started recording. That should make sense.

Another caveat is that recording isn’t free or cheap. There’s a non-trivial amount of overhead involved in keeping track of registers and memory. So use record where it matters.

By default, there’s an upper limit on the number of instructions the record log can contain. This is 200,000 in the default record mode. This can be tweaked however, including setting it to unlimited (which really just means it’ll record until it’s out of memory). See the GDB manual for more info on this.

You can always see what the current record instruction limit is by using the “info record” command.


Reverse debugging is great tool to keep in your toolbox for those tricky bits of code. In the right contexts, it can save you lots of time. Use it judiciously, however. Recording everything in your application wastes memory, memory that your application may actually need. It can also be detrimental to your program’s execution speed.

GDB Tips and Tricks #3: Saving and Restoring Breakpoints Using Files

You spent the last 10 minutes littering your application with breakpoints in all the places where you think bad things might be happening. Then you run your application. Maybe you missed something. Maybe you didn’t. But now after a few minutes of debugging, both your application and GDB appear to be in a funky state. What you want to do is just quit out of everything and start fresh with a new session. But what about all those breakpoints? Typing in the “b” commands was such a chore. Even if you could remember where they all were, the mere thought of doing so is exhausting. And what if you need to start over yet again? Surely there’s a better way.

Did you know you could save your breakpoints to a file? All you need to do is issue the “save breakpoints” command like so…

(gdb) save breakpoints my.brk
Saved to file 'my.brk'.

This will save all types of breakpoints (regular breakpoints, watchpoints, and catchpoints) to the file my.brk. You can in fact name the file whatever you’d like.

Later when you’re ready to reload your breakpoints, you can issue the source command.

(gdb) source my.brk

There’s nothing special about this breakpoints file. It’s just a text file containing a list of gdb commands separated by newlines. In fact, not only can you amend it manually with more breakpoint commands, but you can add in just about any other gdb command in the process.

It’s worth noting that the “source” command isn’t actually specific to breakpoints. It will execute anything in the specified file as written, which makes the “source” command a handy tool in its own right.

GDB Tips and Tricks #2: Setting Breakpoints with Regular Expressions

When debugging applications under gdb, the meek and mighty breakpoint is the cornerstone of just about everything we do. Breakpoints give us the ability to freeze time so we can inspect, and sometimes modify, the world in which our code is running. They can be used to “hit count” a body of count. You can even use breakpoints to automatically run commands within gdb.

So it’s no surprise that “break, or “b”, is one of the first few gdb commands we learn about. The problem with the “break” command, however, is that it allows you to set one breakpoint at a time (That is, unless you specify an overloaded function name as your break location. You’ll end up with multiple breakpoints in that case). If you need to set a lot of breakpoints, “break” can get very repetitive very fast.

Enter “rbreak”, or “rb”, to the rescue. The gdb “rbreak” command allows you to use grep-style regular expressions to set breakpoint locations. As an example, consider the following test fixture class.

struct TestFixture
    void setUp() { std::cout << "Method setUp()\n"; }
    void tearDown() { std::cout << "Method tearDown()\n"; }
    void testA() { std::cout << "Method testA()\n"; }
    void testB() { std::cout << "Method testB()\n"; }
    // ...

Let’s suppose that we want to add breakpoints on only the methods that start with “test”.

(gdb) rb TestFixture::test.*
Breakpoint 1 at 0x4008f2: file gdbtest.cpp, line 7.
void TestFixture::testA();
Breakpoint 2 at 0x400910: file gdbtest.cpp, line 8.
void TestFixture::testB();
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004008f2 in TestFixture::testA() at gdbtest.cpp:7
2       breakpoint     keep y   0x0000000000400910 in TestFixture::testB() at gdbtest.cpp:8

This gave us two breakpoints in this example.

What if I want to set breakpoints for every function in a given file? While not really a regular expression, “rbreak” allows you set these like so.

(gdb) rb TestFixture.h:.
Breakpoint 1 at 0x4008b6: file TestFixture.h, line 5.
void TestFixture::setUp();
Breakpoint 2 at 0x4008d4: file TestFixture.h, line 6.
void TestFixture::tearDown();
Breakpoint 3 at 0x4008f2: file TestFixture.h, line 7.
void TestFixture::testA();
Breakpoint 4 at 0x400910: file TestFixture.h, line 8.
void TestFixture::testB();
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004008b6 in TestFixture::setUp() at TestFixture.h:5
2       breakpoint     keep y   0x00000000004008d4 in TestFixture::tearDown() at TestFixture.h:6
3       breakpoint     keep y   0x00000000004008f2 in TestFixture::testA() at TestFixture.h:7
4       breakpoint     keep y   0x0000000000400910 in TestFixture::testB() at TestFixture.h:8

Note the single colon when using file names vs double colons when specifying class names.

There’s even a way to specify that you want breakpoints for all functions everywhere.

(gdb) rbreak .
Breakpoint 5 at 0x4008b6: file TestFixture.h, line 5.
void TestFixture::setUp();
Breakpoint 6 at 0x4008d4: file TestFixture.h, line 6.
void TestFixture::tearDown();
Breakpoint 7 at 0x4008f2: file TestFixture.h, line 7.
void TestFixture::testA();
Breakpoint 8 at 0x400910: file TestFixture.h, line 8.
void TestFixture::testB();
Breakpoint 9 at 0x4007e5: file gdbtest.cpp, line 4.
int main(int, char**);
Breakpoint 10 at 0x400899: file gdbtest.cpp, line 13.
static void _GLOBAL__sub_I_main();
Breakpoint 11 at 0x400865: file gdbtest.cpp, line 13.
static void __static_initialization_and_destruction_0(int, int);
Breakpoint 12 at 0x400640
<function, no debug info> _init;
Breakpoint 13 at 0x400670
<function, no debug info> std::ios_base::Init::Init()@plt;
Breakpoint 14 at 0x400680
<function, no debug info> __libc_start_main@plt;
Breakpoint 15 at 0x400690
<function, no debug info> __cxa_atexit@plt;
Breakpoint 16 at 0x4006a0
<function, no debug info> std::ios_base::Init::~Init()@plt;
Breakpoint 17 at 0x4006b0
<function, no debug info> std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)@plt;
Breakpoint 18 at 0x4006c0
<function, no debug info> __stack_chk_fail@plt;
Breakpoint 19 at 0x4006e0
<function, no debug info> _start;
Breakpoint 20 at 0x400710
<function, no debug info> deregister_tm_clones;
Breakpoint 21 at 0x400750
<function, no debug info> register_tm_clones;
Breakpoint 22 at 0x400790
<function, no debug info> __do_global_dtors_aux;
Breakpoint 23 at 0x4007b0
<function, no debug info> frame_dummy;
Breakpoint 24 at 0x400930
<function, no debug info> __libc_csu_init;
Breakpoint 25 at 0x4009a0
<function, no debug info> __libc_csu_fini;
Breakpoint 26 at 0x4009a4
<function, no debug info> _fini;

This, of course, includes a lot of standard library stuff that you probably don’t care about. So it’s not as interesting as it first might seem.

“rbreak” can save you a lot of time typing and trying to remember exact function names. It’s a super-handy tool to keep in your toolbox.

(p.s., if you accidentally set more breakpoints than you had intended, you can delete a specific breakpoint by using the “delete breakpoint [breakpoint-number]” command. You can get the breakpoint number from “info breakpoints”. To delete all breakpoints, use the “delete breakpoints” command).