GDB Debugging Automation with Python : Implementing a memory leak detector

1. Introduction : In this post , I will talk about how to automate debugging with GDB. And as an example project I will show a memory leak detector. The final leak detector can be seen here in action :

2. About GDB automation : Previously posted a small writing about Windbg: https://nativecoding.wordpress.com/2016/01/10/automate-attach-to-process-on-windows-with-windbg/

Similarly to Windbg , GDB supports script files, which allows you to save a batch of commands. You can also have those in gdb init file to load any time you load GDB. Additionally to that , gdb CLI interface also has a batch mode, therefore you can also automate GDB commands with an external Bash script. Below you can see a simple GDB script that dumps information about malloc calls :

To load a script like above , you just type “source malloc_dumper.txt” in GDB prompt.

However debugging automation can be really powerful when you use GDB`s Python API. Starting from GDB7, GDB comes with an embedded Python interpreter and also exposes a module named “gdb” for use from Python.

3. What can be done with GDB Python API : Firstly note that “gdb” module for Python will only be available when your script executed from GDB as it dynamically injects it to its interpreter. Therefore you will need to directly work with GDB prompt to explore it. Initially. Here is a list APIs :

https://sourceware.org/gdb/onlinedocs/gdb/Python-API.html#Python-API

To summarise what can be done generally :

  • You can do anything you could do with GDB scripts simply by doing “gdb.execute(“gdb_command_you_want_to_enter”)
  • You can create new GDB commands or  functions
  • You can create pretty printers. ( See the final Links section for an STL pretty printer example )
  • You can access breakpoints, frames & blocks and symbols, processs , threads and exceptions. values and more and all of these are provided as classes which makes job of automation very convenient compared to simple GDB scripting

4. Example project “memory leak detector” : I coded a small Python GDB extension script which dumps information about malloc,realloc, calloc and free calls from GNU LibC runtime. How it works briefly :

  • Places breakpoints for GNU LibC runtime memory functions. Also places breakpoints for main and exit function to detect start and the end of the session.
  • When a memory-function related breakpoint hit , it takes the control , captures the arguments passed to the function, captures callstack, and executes it until the end of its frame in order to capture the return value and then continues debugging
  • Also created a small Bash script , which makes it easy to use “memdump” extension. It basically executes GDB in batch mode, loads the Python script to GDB`s memory and executes it.

Note : As prerequisites you will need to install debug version of GNU LibC runtime. On Ubuntu :

sudo apt-get install libc6-dbg

And on CentOS :

yum install yum-utils
debuginfo-install glibc

And here you can see the Python implementation :

You can use the command below in order to start a GDB session by loading memdump.py :

gdb -batch -ex “source memdump.py” -ex ‘memdump’ -ex ‘r’ <debugee_execitable>

5. Analysing dump output : The previous GDB extension creates a text file with all memory operations information. I also implemented a separate Python script that parses the GDB extension`s output and finds out leaks. Basically the way it works :

  • For each calloc and malloc we add the memory event to a hash table by making the memory address key value
  • For each realloc , we remove the entry in the hash table for a previously allocated address and add a new entry with new memory address
  • For each free operation, we remove the entry from the dictionary.
  • Specifically to GNU LibC Runtime, we ignore memory operations which belong to directly GNU LibC Runtime`s internal functions
  • Finally each entry remaining in the dictionary gives us leaks.

Here is the analyser script :

6. Links : Here you can find a list of nice resources for the topic :

A presentation about GDB Python extensions : https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2011/GdbPythonPresentation/GdbPython.html#1

A pretty printing example : http://hgad.net/posts/object-inspection-in-gdb/

A Python extension to make deadlock analysis easier : http://www.linuxjournal.com/article/11027?page=0,0

Another deadlock detector :

https://github.com/xmementoit/gdb-automatic-deadlock-detector/blob/master/gdbDisplayLockedThreads.py

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s