GDB Tips and Tricks #2: Setting Breakpoints with Regular Expressions

When debugging applications under gdb, the meek and mighty breakpoint is the cornerstone of just about everything we do. Breakpoints give us the ability to freeze time so we can inspect, and sometimes modify, the world in which our code is running. They can be used to “hit count” a body of count. You can even use breakpoints to automatically run commands within gdb.

So it’s no surprise that “break, or “b”, is one of the first few gdb commands we learn about. The problem with the “break” command, however, is that it allows you to set one breakpoint at a time (That is, unless you specify an overloaded function name as your break location. You’ll end up with multiple breakpoints in that case). If you need to set a lot of breakpoints, “break” can get very repetitive very fast.

Enter “rbreak”, or “rb”, to the rescue. The gdb “rbreak” command allows you to use grep-style regular expressions to set breakpoint locations. As an example, consider the following test fixture class.

struct TestFixture
{
    void setUp() { std::cout << "Method setUp()\n"; }
    void tearDown() { std::cout << "Method tearDown()\n"; }
    void testA() { std::cout << "Method testA()\n"; }
    void testB() { std::cout << "Method testB()\n"; }
    // ...
};

Let’s suppose that we want to add breakpoints on only the methods that start with “test”.

(gdb) rb TestFixture::test.*
Breakpoint 1 at 0x4008f2: file gdbtest.cpp, line 7.
void TestFixture::testA();
Breakpoint 2 at 0x400910: file gdbtest.cpp, line 8.
void TestFixture::testB();
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004008f2 in TestFixture::testA() at gdbtest.cpp:7
2       breakpoint     keep y   0x0000000000400910 in TestFixture::testB() at gdbtest.cpp:8

This gave us two breakpoints in this example.

What if I want to set breakpoints for every function in a given file? While not really a regular expression, “rbreak” allows you set these like so.

(gdb) rb TestFixture.h:.
Breakpoint 1 at 0x4008b6: file TestFixture.h, line 5.
void TestFixture::setUp();
Breakpoint 2 at 0x4008d4: file TestFixture.h, line 6.
void TestFixture::tearDown();
Breakpoint 3 at 0x4008f2: file TestFixture.h, line 7.
void TestFixture::testA();
Breakpoint 4 at 0x400910: file TestFixture.h, line 8.
void TestFixture::testB();
(gdb) info breakpoints
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00000000004008b6 in TestFixture::setUp() at TestFixture.h:5
2       breakpoint     keep y   0x00000000004008d4 in TestFixture::tearDown() at TestFixture.h:6
3       breakpoint     keep y   0x00000000004008f2 in TestFixture::testA() at TestFixture.h:7
4       breakpoint     keep y   0x0000000000400910 in TestFixture::testB() at TestFixture.h:8

Note the single colon when using file names vs double colons when specifying class names.

There’s even a way to specify that you want breakpoints for all functions everywhere.

(gdb) rbreak .
Breakpoint 5 at 0x4008b6: file TestFixture.h, line 5.
void TestFixture::setUp();
Breakpoint 6 at 0x4008d4: file TestFixture.h, line 6.
void TestFixture::tearDown();
Breakpoint 7 at 0x4008f2: file TestFixture.h, line 7.
void TestFixture::testA();
Breakpoint 8 at 0x400910: file TestFixture.h, line 8.
void TestFixture::testB();
Breakpoint 9 at 0x4007e5: file gdbtest.cpp, line 4.
int main(int, char**);
Breakpoint 10 at 0x400899: file gdbtest.cpp, line 13.
static void _GLOBAL__sub_I_main();
Breakpoint 11 at 0x400865: file gdbtest.cpp, line 13.
static void __static_initialization_and_destruction_0(int, int);
Breakpoint 12 at 0x400640
<function, no debug info> _init;
Breakpoint 13 at 0x400670
<function, no debug info> std::ios_base::Init::Init()@plt;
Breakpoint 14 at 0x400680
<function, no debug info> __libc_start_main@plt;
Breakpoint 15 at 0x400690
<function, no debug info> __cxa_atexit@plt;
Breakpoint 16 at 0x4006a0
<function, no debug info> std::ios_base::Init::~Init()@plt;
Breakpoint 17 at 0x4006b0
<function, no debug info> std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)@plt;
Breakpoint 18 at 0x4006c0
<function, no debug info> __stack_chk_fail@plt;
Breakpoint 19 at 0x4006e0
<function, no debug info> _start;
Breakpoint 20 at 0x400710
<function, no debug info> deregister_tm_clones;
Breakpoint 21 at 0x400750
<function, no debug info> register_tm_clones;
Breakpoint 22 at 0x400790
<function, no debug info> __do_global_dtors_aux;
Breakpoint 23 at 0x4007b0
<function, no debug info> frame_dummy;
Breakpoint 24 at 0x400930
<function, no debug info> __libc_csu_init;
Breakpoint 25 at 0x4009a0
<function, no debug info> __libc_csu_fini;
Breakpoint 26 at 0x4009a4
<function, no debug info> _fini;

This, of course, includes a lot of standard library stuff that you probably don’t care about. So it’s not as interesting as it first might seem.

“rbreak” can save you a lot of time typing and trying to remember exact function names. It’s a super-handy tool to keep in your toolbox.

(p.s., if you accidentally set more breakpoints than you had intended, you can delete a specific breakpoint by using the “delete breakpoint [breakpoint-number]” command. You can get the breakpoint number from “info breakpoints”. To delete all breakpoints, use the “delete breakpoints” command).

GDB Tips and Tricks #1: A Tale of Two Terminals

It’s occasionally handy to be able to separate an application’s console I/O from that of gdb. Perhaps you’re debugging a particularly chatty application that’s preventing you from seeing what’s going on with gdb. Or maybe you’re debugging an ncurses application, a situation where gdb’s terminal I/O can actually interfere with the application’s UI.

Fortunately, there’s a mechanism that allows your application’s console I/O to be handled in a completely different terminal than that of gdb. The gdb command is called “tty” (not to be confused with the command line tool “tty” which we’ll also be using here).

Step 1: Figure out the device file for the terminal where you want your app’s console I/O to be handled. In that terminal…

skirk@dormouse:~$ tty
/dev/pts/18

Note: It probably goes without saying, but the device path will vary from terminal to terminal.

Step 2: Put the shell in that terminal to sleep so it’s not competing with your application for I/O.

skirk@dormouse:~$ sleep 1000000

Step 3: In a second terminal, fire up gdb with your application.

skirk@dormouse:~$ gdb ./myapp
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ttytest...done.
(gdb)

Step 4: Within gdb, set the device file path for terminal where your app’s I/O should be handled.

(gdb) tty /dev/pts/18

Step 5: Debug your app! Note: You may see a warning about GDB not being able to set the controlling terminal. This is safe to ignore.

The Path to Part 107

Being a multicopter enthusiast, it’s not uncommon for family and friends to ask me for help in taking aerial photos. I’m usually happy to oblige. Sometimes, however, these requests have some sort of commercial aspect to them. Maybe they’re trying to sell their house. Or maybe the company they work for is having an open house on a new facility they just built and wants some fun marketing photos. In the past, I’ve always had to turn those types of requests down. Not only did I not have the appropriate gear (that FlyCam2 velcroed to the top of my 250 just doesn’t cut it), but I also wasn’t legally allowed to do so.

I recently decided to solve both of those problems. Solving the equipment problem was easy (I bought a DJI Phantom 4 Advanced). The legality problem, however, required a bit more effort.

Since August 2016, the FAA has required commercial drone pilots to be certificated under the rules set forth in what’s commonly referred to as CFR 14 Part 107. “CFR” is the Code for Federal Regulations. It’s a set of rules for all sorts of things including elections, food and drugs, wildlife management, and even the Executive Office of the U.S. President. CFR Title 14 covers Aeronautics and Space (aka aviation regulations) and Part 107 is the section specific to “Small Unmanned Aircraft Systems”.

Part 107 not only spells out the rights and responsibilities of the sUAS remote pilot in command, but it also specifies all the things that remote PICs need to know in order to become certificated for commercial operations. It’s a LOT of information. And it’s been criticized for containing material that many folks feel is irrelevant for sUAS operations. But it is what it is. If we want to make money with our toys, we have to play by the rules. For me, that meant I had to take an aeronautical knowledge test.

For someone licensed to fly a manned aircraft, passing the Part 107 aeronautical knowledge test should be a breeze. It’s mostly a subset of information you already need to know to obtain or maintain your pilot’s license. My only experience with planes, however, was as a passenger. The test material was entirely new to me. But I did it on my first try. And if I can pass the test, so can you.

Study Materials

In preparing for the exam, I wasn’t really sure where to start. I Googled around and bounced from site to site that offered loads of study tips and links to YouTube’s videos. And it didn’t take long for me to become overwhelmed. I decided that I needed something a bit more structured. So I signed up for an online course, as well as picked up a Part 107 study guide from Amazon.

The course I signed up for was Drone Pilot Ground School. It’s a little pricey compared to some other options ($299 at the time of this writing). But for me, it was the Munchkin City on my path to Oz. It’s well laid out, provides downloadable videos of the lectures, PDF transcripts, quizzes, practice tests, and even bonus material on drone business related topics – all of which I found extremely helpful. It also provides you with access to material for 5 years. Certificated sUAS operators need to retest every 2 years, so I will most certainly be repeating this course.

Once I got about ⅔ of the way through the course, I started working my way through the book/study guide, “Remote Pilot Test Prep – UAS: Study & Prepare”. Much of what was in the book reinforced what I had already learned in the online course, but there were also a few new nuggets here and there. Another nice thing about this book is that it gives you access to 5 online practice tests.

Even with these two sources of study material, I still ended up feeling shaky on a few topics (weather, METARs, and airport signage, mostly). So it was then that I turned back to YouTube to fill in the gaps. After having gone through both the online course and the study guide, it was much easier to wade through all the Part 107 related YouTube videos and select the content I needed.

I put in around 50 hours of total study time. That’s around 1-2 hours of studying a day for about a month. After all that cramming, I felt it was time to pull the trigger on the test. I wasn’t 100% confident that I would actually pass, mind you. But I was 100% confident that my brain was fully saturated with information. It could hold no more. 🙂

Test Registration

The FAA publishes a list of testing centers across the U.S. So the first thing I did was phone up the testing center nearest me and asked to register for the test. “We only administer the tests. We don’t handle registration here. You need to contact CATS.”

Oops.

I had ignored the phone numbers for both CATS and PSI at the top of the FAA’s testing center list, mostly because I had no idea what their relevance was to me or the testing centers.

I visited the CATS website first. I had hoped to register for the test electronically. That way I could take my time and make sure all of my information was entered correctly. At the time of this writing, the CATS website provides a lot of information but no way to register for the test. You have to call them up and give a person your info over the phone.

So I called them up. And then I waited on hold for a crazy long time (like, cable-company hold time). Eventually, someone came on the line. I explained what I wanted to do, and they started taking my information. The CATS person stressed how important it was that the information I provided them was exact, right down to whether my driver’s license used “RD” vs “ROAD” in my address. I paid $150 for the exam. And then I was asked which testing center I wanted to visit. I told her. The lady on the phone said, “OH! I should have asked that first. I registered you through PSI and that particular facility uses CATS. I’m going to have to refund you, re-enter your registration information, and then rebill you via CATs.” I was then put on hold for another long while as she did all of that.

Eventually, she came back on the phone and asked when I wanted to take the exam. She’d need to call the facility and find out what availability they had. Again, I was put on hold. Once she returned to the call, she said that everything was all set and that I should be receiving a confirmation email shortly with all of the details. Hooray!

After I hung up with her, I quickly received a confirmation email. Upon scanning the details, I noticed that “LANE” in my street address was misspelled as “LANAE”. So much for accuracy. But I guess the burden was on me to make sure my information was correct. So I called them back and waited on hold again.

Once I was able to talk to someone again, I explained what happened. The correction was made and I received a new confirmation email.

The Test

A few days later, I drove to Maine Instrument Flight, Inc. in Augusta to take the exam. Maine Instrument Flight provides a number of services, among them being a flight school. I walked up to the main desk and announced that I was there to take the Remote Pilot general knowledge test. A very nice lady whose name escapes me checked for my name on the test schedule and then lead me to the second floor.

We sat together at a desk, where she took my ID and verified all of my info in the CATS system. She again stressed how important it was that my information was exact.

After being told to leave my cellphone and anything else electronic behind, she lead me into a small room that contained three desks facing the wall, each separated by a cubicle partition. All of the desks had the same arrangement of computers, scratch paper, calculators, and pencils.

She sat down behind a desk and attempted to walk me through a demo of the CATS software. As soon as she showed me how to change the font size, the software crashed. As she restarted everything, I told her not to worry about a demo and that I’d just figure it out. The software looked remarkably similar to the study guide practice exams.

She handed me a copy of Airman Knowledge Testing Supplement and a few pieces of clear plastic for reasons that aren’t entirely clear to me (I think she was afraid I’d accidentally mark in the book). She explained that there was a camera watching the room and that when I was finished, I should just stand up and poke my head out the door. Then she left me to it. At that point, I clicked the start button and the countdown clock began ticking away.

The test itself was a lot more straightforward than I expected. It’s 60 questions and you’re given 2 hours to complete it. The questions were very similar to the practice tests I had taken. I was very happy that I had spent so much time going over sectional charts as it seemed at least ¼ of the questions revolved around them in some way. It only took me around 30 minutes to complete the exam.

I stood up, poked my head through the door, and then the lady reappeared. She lead me back to the original desk. She punched a few buttons and then she printed out my results. 88%! Woohoo! She asked if I wanted to see the questions I missed. “Sure!” “But it doesn’t tell you the correct answer,” she said. “Meh. Nevermind then.” I signed something, received my results which contained by test ID along with a page that described the next steps, picked up the rest of my belongings, and was on my way.

IACRA

Your work isn’t finished once you pass the aeronautical test. After you pass the exam, you need to submit an application to the FAA for a Remote Pilot Certificate. This whole process is managed through a system called IACRA (Integrated Airman Certification and Rating Application). Before you can do so, however, you have to wait up to 48 hours for your test results to show up in IACRA’s system. It was Memorial Day weekend at the time, so I figured I’d kick back and give the whole process a full three days. I wasn’t in a terrible hurry.

When Monday rolled around, I created my IACRA account. I then filled out an application for a remote pilot certificate. As part of the process, you have to lookup your test results using the code appearing on your results document and “attach” it to the application. Once it’s submitted, you can even download a PDF copy of the completed application. The PDF is a little confusing, though. Even though your test results are an attachment of your application, they don’t appear in the download. For a long while, I questioned whether or not I had missed a button or a checkbox in the application process.

Temporary Certificate

After about a week, my temporary airman certificate was available for download. The temporary certificates aren’t much to look at. They don’t have an airman number, a certification number, or much else. It’s mostly a cookie cutter form that’s autopopulated with your name and address and that’s about it. But it does allow you to start performing commercial operations with your drone.

Final Certificate

It took two months (May 29th to July 28th) from the time that my application was submitted to receive my final certificate. I was expecting a colorful certificate that I could hang on my wall. Or maybe a paper card similar to what recreational drone operators are supposed to carry around with their registered drones. I was pleasantly surprised at what did show up. It was a little blue/green plastic card similar in size to a driver’s license. It had all my information on it as well as a really cool FAA hologram. Very fancy. It will undoubtedly impress all my nerd friends.

Conclusion

All in all the Part 107 certification process is pretty straightforward. The hardest part, IMO, was the waiting. I was also unimpressed by CATS, but the systems the FAA employs for registration and application are all easy to use and well documented.

The whole certification experience has since inspired me to start my own aerial photography business – Kirk Aerial Solutions, LLC. If you’ve gone through the process, I’d love to hear how your experience was different or similar to my own. Leave a comment below and tell me about it. And if you’re just starting your journey towards Part 107 certification, I hope hearing about my own experience helps in some way.

-Shane