GDB Tips and Tricks #6: Examining Data Types

It’s often the case that while stepping through code you encounter a variable that’s unknown to you. If it doesn’t appear to be relevant to the task at hand, perhaps you choose to ignore it. But if it may have some significance to the problem you’re trying to solve, you’ll obviously need to learn a bit more about it.

The first thing you might want to learn about the variable is its type. You can turn to the source code to seek this out. If the type turns out to be a struct/class, typedef, or type alias, you might need to do a bit more spelunking to fully understand what kind of data is in play.

Fortunately, gdb provides us with a couple of commands to help us out when it comes to data types – whatis and ptype

whatis

The whatis command tells us the superficial type of a variable or expression. By that I mean, it will show the type of something as it appears in code. Things like typedefs or type aliases aren’t “unrolled.” But the whatis command accepts expressions too. So if you’d like unroll a typedef or type alias, you can do so manually. Let’s see some examples.

Let’s use the following snippet of code.

struct MyStruct
{
    int x;
    MyStruct() : x(2) {}
};
 
using _IntegerValue = int;
using IntValueAlias = _IntegerValue;
 
int main()
{
    int a = 0;
    IntValueAlias b = 1;
    MyStruct c;
    return 0;
}

After we compile, launch gdb, and step into main, let’s ask gdb to tell us the types of the variables a, b, and c.

(gdb) whatis a
type = int
(gdb) whatis b
type = IntValueAlias
(gdb) whatis c
type = MyStruct

As you’d probably expect, gdb prints “int” when we ask about variable a’s type. When we ask for the type of variable b, we see “IntValueAlias”. You’ll note from the code snippet above that IntValueAlias is just a type alias for another type alias which isn’t shown in gdb’s output. When asked about the type of c, we get the struct name, “MyStruct”.

If we want to dig a bit further on the type alias, we can use whatis like so.

(gdb) whatis IntValueAlias
type = _IntegerValue
(gdb) whatis _IntegerValue
type = int

If you expect whatis to give you more detailed information about structures and classes, you’ll be disappointed to know that it doesn’t help you out. The documentation for whatis actually claims that the /M option will show class methods and that /T shows typedefs defined in a class, but I’ve never been able to get these options to do anything.

(gdb) whatis MyStruct
type = MyStruct

I should mention a couple of things about templates and whatis. The whatis command substitutes template parameters and typedefs when displaying type information for variables of template types. This means you can see pretty nasty stuff.. For example, take this snippet of code.

int main()
{
    std::string a;
    std::map<std::string, std::string> b;
    return 0;
}

If we ask gdb what the type of a is, we’ll see the following.

(gdb) whatis a
type = std::__cxx11::string

That’s not exactly what we typed in our code, but it’s workable.

What about the type of b?

(gdb) whatis b
type = std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >

As my two-year old daughter would say, “YUCK!”. Unfortunately, it is what it is and we have to work with it.

ptype

The ptype command is a bit more heavy duty than whatis. Think of it as whatis on steroids. In the case of typedefs and type aliases, ptype always unrolls them. In the case of structs and classes, ptype gives much more detailed information, which includes methods and member variables. In the case of templates, well, we get a lot more yuck.

Let’s look again at our first example.

struct MyStruct
{
    int x;
    MyStruct() : x(2) {}
};
 
using _IntegerValue = int;
using IntValueAlias = _IntegerValue;
 
int main()
{
    int a = 0;
    IntValueAlias b = 1;
    MyStruct c;
    return 0;
}

Just as we did before, let’s examine the types of variables a, b, and c. But instead of using whatis, let’s use ptype.

(gdb) ptype a
type = int
(gdb) ptype b
type = int
(gdb) ptype c
type = struct MyStruct {
    int x;
  public:
    MyStruct(void);
}

The type of a is shown as int, just as before. The type of b has been unrolled all the way down to the type of the original type alias, which is int. The type of c now shows the layout of the MyStruct struct.

I’ll spare you from a ptype example using templates. As you can imagine, a lot more information is displayed as compared to whatis.

Conclusion

Both whatis and ptype are handy tools for inspecting data types. You could certainly accomplish the same types of things manually within your IDE. But these commands allow you to keep your focus on the debugging session instead of your IDE, which hopefully means being more efficient in tracking down the source of bugs.

GDB Tips and Tricks #5: The Display Command

Once of the cool things about debugging with IDEs is that they typically give you a nice mechanism to watch the changing state of variables as you step through code. In Visual Studio, for example, you can right click on a variable name and select “Add Watch” from the menu. The variable name and its current value will be shown in a little “Watch” window. You can watch as many variables as you have the resources and patience for. As you step through the code, anytime the value of a watched variable changes, that change is reflected in the Watch window.

Can we do something similar in gdb? Absolutely.

The command we’re interested in is display. When gdb is told to display a variable, it’ll report that variable’s current value every time program execution pauses (e.g., stepping through the code).

Let’s see an example using the following snippet of code.

// demo.cpp
int main()
{
    int a = 1;
    int b = 2;
    int c = 3;
 
    a = a + 1;
    b += a;
    c = a * b + c;
 
    return 0;
}

First we compile and then launch gdb.

skirk@dormouse:~$ g++ -g ./demo.cpp -o demo
skirk@dormouse:~$ gdb ./demo
GNU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/skirk/demo...done.
(gdb)

Let’s now run our app, stopping at main().

(gdb) start
Temporary breakpoint 1 at 0x4004bb: file ./demo.cpp, line 3.
Starting program: /home/skirk/demo 
 
Temporary breakpoint 1, main () at ./demo.cpp:3
3	    int a = 1;

Let’s say at this point we want to display the values of variables a, b, and c as we step through the code. We can issue the display command like so.

(gdb) display a
1: a = 32767
(gdb) display b
2: b = 0
(gdb) display c
3: c = 0

After each display is executed, gdb shows the current value for the variable specified. In this example, our variables have bogus values because they haven’t been initialized yet. Let’s now step through the code and see what display does for us.

(gdb) n
4	    int b = 2;
1: a = 1
2: b = 0
3: c = 0
(gdb) n
5	    int c = 3;
1: a = 1
2: b = 2
3: c = 0
(gdb) n
7	    a = a + 1;
1: a = 1
2: b = 2
3: c = 3
(gdb) n
8	    b += a;
1: a = 2
2: b = 2
3: c = 3
(gdb) n
9	    c = a * b + c;
1: a = 2
2: b = 4
3: c = 3
(gdb) n
11	    return 0;
1: a = 2
2: b = 4
3: c = 11

Every time we step through the code, our program execution pauses and the current values of the variables we asked gdb to display are shown. As you can imagine, this can save a tremendous amount of time over, say, repeatedly using a step command followed by a print command.

It’s worth noting that the variable value information will only be displayed for variables that are currently in scope. If the variables we’re interested in are local to a function that at some point returns, those variables will no longer be displayed once they’re out of scope. However, gdb doesn’t forget about those variables. It will absolutely display them the next chance it gets. If you later step into that function again, those variables will be displayed.

When you’re no longer interested in a given variable, you can issue the undisplay command. The gotcha here is that undisplay doesn’t operate on variable names. It operates on display numbers. “Where is the display number?” you ask. It’s the number next to the “variable=value” line in the display output. In our example above, the display output for our variable c is “3: c = 11”. Note the 3 before the colon? It’s not just to pretty up the output. That’s the display number assigned to that particular variable.

You can undisplay a single display number like so.

(gdb) undisplay 1

You can also undisplay multiple display numbers at once.

(gdb) undisplay 2 3

Note that the display command shouldn’t be confused with the watch command, which serves a related purpose. The watch command works more like a smart breakpoint (these are actually called watchpoints) in that it stops program execution and displays a given variable’s value only when the value changes. The display command provides a continuous display of variables and doesn’t affect program execution at all.

GDB Tips and Tricks #4: Reverse Debugging

How many times have you stepped through code only to find that you’ve gone too far? Maybe you overstepped a critical point in the code’s execution. Or perhaps you stepped over a function you intended to step into.

Did you know that GDB could actually step backwards through of code? That’s right. You can have GDB go back in time. This is often referred to as “reverse debugging.” But how does it work?

How It Works

Reverse debugging actually relies upon another gem in GDB’s bag-of-tricks called “process record and replay”. Rolls right off the tongue doesn’t it? I won’t spend a lot of time going into the details of PRR here, but it’s quite powerful. The only PRR command we need to be concerned with in this discussion is “record”.

The “record” command begins recording the execution of your application, making notes of things like memory and register values. When you arrive at a point in your application at which you’d like to go backwards, you can issue the “reverse” versions of all the navigation commands you’re already familiar with. This include reverse-step (rs), reverse-next (rn), reverse-continue (rc), and reverse-finish (no short version 🙁 ). As you move backwards through code, gdb reverts the state of memory and registers, effectively unexecuting lines of code.

Let’s see an example using the code snippet below.

#include <iostream>
 
int sum(int a, int b)
{
    int result = a + b;
    return result;
}
 
int main(int argc, char **argv)
{
    int a = 12;
    int b = 13;
    int c = sum(a, b);
 
    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";
 
    return 0;
}

Compile this (don’t forget to compile it with the ‘-g’ flag!) and fire up gdb. Then set a breakpoint at main. We can’t begin recording program execution before it’s actually running. So we issue the run command, which will execute our application and promptly break at main.

(gdb) break main
Breakpoint 1 at 0x4007df: file gdbtest.cpp, line 11.
(gdb) run
Starting program: /home/skirk/gdbtest 
 
Breakpoint 1, main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:11
11	    int a = 12;

At this point, we issue the “record” command to begin recording.

(gdb) record

Now let’s start stepping through the code.

(gdb) n
12	    int b = 13;
(gdb) n
13	    int c = sum(a, b);
(gdb) n
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

We’re now at the point just before the sum is written to stdout. What if I had intended to step into the sum function to see what it’s doing? Let’s back up to just before the sum function is called and then step into it.

(gdb) reverse-next
13	    int c = sum(a, b);
(gdb) s
sum (a=12, b=13) at gdbtest.cpp:5
5	    int result = a + b;

Now we appear to have gone back into time. This allows us to step into the sum function. At this point, we can inspect the values of parameters a and b as we normally would.

 
(gdb) print a
$1 = 12
(gdb) print b
$2 = 13

If we’re satisfied with the state of things, we can allow the program to continue on.

(gdb) c
Continuing.
 
No more reverse-execution history.
main (argc=1, argv=0x7fffffffdf28) at gdbtest.cpp:15
15	    std::cout << "The sum of " << a << " and " << b << " is " << c << "\n";

An interesting thing happened here. The program execution stopped at the point at which we previously started stepping backwards. When stepping through code using recorded history, “continue” will continue program execution until the history has been exhausted, unless, of course, it has some other reason to stop such as breakpoints and the like.

Let’s now stop the recording process using the “record stop” command and allow the program to continue execution until completion.

(gdb) record stop
Process record is stopped and all execution logs are deleted.
(gdb) c
Continuing.
The sum of 12 and 13 is 25
[Inferior 1 (process 10608) exited normally]
(gdb)

Gotchas
What if we hadn’t stopped recording? Well, it depends. If your version of the runtime executes instructions that aren’t supported by PRR, then you may encounter errors such as this…

Process record does not support instruction 0xc5 at address 0x7ffff7dee8b7.
Process record: failed to record execution log.
 
Program stopped.
_dl_runtime_resolve_avx () at ../sysdeps/x86_64/dl-trampoline.h:81
81	../sysdeps/x86_64/dl-trampoline.h: No such file or directory.

In this case, AVX instructions are being executed which aren’t supported by the record process. (In this particular case, there’s a workaround. We can export the environment variable LD_BIND_NOW=1 which resolves all symbols at load time. Doing so actually prevents the call to _dl_runtime_resolve_avx later.)

It’s also possible you might see something like…

The sum of 12 and 13 is 25
The next instruction is syscall exit_group.  It will make the program exit.  Do you want to stop the program?([y] or n)

Here you’re prompted as to whether or not you want to stop the program. Regardless of what you choose, you’re still able to navigate backwards in program execution. That’s right – you can reverse debug an application that has finished running.

Caveats

There are a few caveats when performing reverse debugging.

The first is that you can’t move backwards beyond the point at which you started recording. That should make sense.

Another caveat is that recording isn’t free or cheap. There’s a non-trivial amount of overhead involved in keeping track of registers and memory. So use record where it matters.

By default, there’s an upper limit on the number of instructions the record log can contain. This is 200,000 in the default record mode. This can be tweaked however, including setting it to unlimited (which really just means it’ll record until it’s out of memory). See the GDB manual for more info on this.

You can always see what the current record instruction limit is by using the “info record” command.

Conclusion

Reverse debugging is great tool to keep in your toolbox for those tricky bits of code. In the right contexts, it can save you lots of time. Use it judiciously, however. Recording everything in your application wastes memory, memory that your application may actually need. It can also be detrimental to your program’s execution speed.