Finding Memory leaks Within Solaris Applications Using libumem

Debugging Methodology :

The malloc() and free() memory management methods are used by many application developers. An application can be written without a dependence on any particular memory management programming interface by using the standard memory management methods malloc() and free(). This section will outline the steps needed to take advantage of the libumem library to debug an application's memory transactions.

Library Interposition and libumem Flags

If the libumem library is interposed (by setting the LD_PRELOAD environment variable) when executing an application, the malloc() and free() methods defined within the libumem library will be used whenever the application calls malloc() or free(). In order to take advantage of the debugging infrastructure of the libumem library, one needs to set the UMEM_DEBUG and the UMEM_LOGGING flags in the environment where the application is being executed.  The most common values for these flags are as follows: UMEM_DEBUG=default and UMEM_LOGGING=transaction.  With these settings, a thread ID, high-resolution time stamp, and stack trace are recorded for each memory  transaction initiated by the application.

The following are examples of the commands used to set the appropriate debug flags and interpose the libumem library when executing an application.


(csh)

%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;
setenv LD_PRELOAD libumem.so.1;)

or

(bash)

bash-2.04$UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1

More details about the debug flags (UMEM_DEBUG and UMEM_LOGGING) can be found in the umem_debug(3MALLOC) man page.

MDB Commands

The developer can view the debug information pertaining to an application's memory management transactions by using MDB.  The following commands within MDB can be used to provide a great deal of information about the memory transactions that took place during the execution of the application.

::umem_status

    * Prints the status of the umem indicating if the logging features have been turned on or off

::findleaks

    * Prints a summary of the memory leaks found within the application
::umalog

    * Prints the memory transactions initiated by the application and the correlated stack traces
::umem_cache

    * Prints the details about each of the umem caches
[address]::umem_log

    * Prints the umem transaction log for the application
[address]::umem_verify

    * Prints the integrity of the umem caches which is useful in determining if a buffer has been corrupted

address$<bufctl_audit

    * Prints the contents of the umem_bufctl_audit structure as defined in the /usr/include/umem_impl.h header file

Example :

Traditional Memory Leak

In order to examine if SunMC has a memory leak, one can execute the following steps to narrow down the section of the code which is causing the leak.

1. The libumem library is only available on systems which are running the Solaris 9 OS, Update 3 and above.
%uname -a
SunOS hostname 5.9 Generic_112233-05
(csh)

2. Execute the application with the libumem library interposed and the appropriate debug flags set.
%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction;setenv LD_PRELOAD libumem.so.1; ./a.out)
(bash)
#UMEM_DEBUG=default;UMEM_LOGGING=transaction;LD_PRELOAD=libumem.so.1; ./a.out
3. Use the gcore (1) command to get an application core to analyze the application's memory transactions.

%ps -ef | grep esd
user1     970   714  0 10:42:42 pts/4    0:00 ./a.out

%gcore 970
gcore: core.970 dumped
4. Use MDB to analyze the core for memory leaks using the commands described in the previous section.

$mdb core.970
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]

> ::umem_log
CPU ADDR     BUFADDR         TIMESTAMP THREAD  
  0 0002e0c8 00055fb8     159d27e121a0 00000001
  0 0002e064 00055fb8     159d27e0fce8 00000001
  0 0002e000 00049fc0     159d27da1748 00000001
    00034904 00000000                0 00000000
    00034968 00000000                0 00000000
    ... snip ...
 
Here we can see that there have been three transactions by thread #1 on cpu #0.

> ::umalog

T-0.000000000  addr=55fb8  umem_alloc_32
         libumem.so.1`umem_cache_free+0x4c
         libumem.so.1`process_free+0x68
         libumem.so.1`free+0x38
         main+0x18
         _start+0x108

T-0.000009400  addr=55fb8  umem_alloc_32
         libumem.so.1`umem_cache_alloc+0x13c
         libumem.so.1`umem_alloc+0x44
         libumem.so.1`malloc+0x2c
         main+0x10
         _start+0x108

T-0.000461400  addr=49fc0  umem_alloc_24
         libumem.so.1`umem_cache_alloc+0x13c
         libumem.so.1`umem_alloc+0x44
         libumem.so.1`malloc+0x2c
         main+4
         _start+0x108
 The three transactions consist of one allocation to the 24 byte umem
cache, and one memory allocation and release from the 32 byte umem
cache. Note that the high resolution timestamp output in the upper left
hand corner is relative to the last memory transaction initiated by the
application.

> ::findleaks
CACHE     LEAKED   BUFCTL CALLER
0003d888       1 00050000 libumem.so.1`malloc+0x0
----------------------------------------------------------------------
 Total       1 buffer, 24 bytes

  This shows that there is one 24 byte buffer which has been leaked.