README.md 5.61 KB
Newer Older
Timo Bingmann's avatar
Timo Bingmann committed
1
# README for malloc_count #
Timo Bingmann's avatar
Timo Bingmann committed
2

3
`malloc_count` provides a set of source code tools to **measure the amount of
4
5
6
7
8
9
10
11
12
13
14
allocated memory of a program at run-time**. The code library provides
facilities to

* measure the **current and peak** heap memory allocation, and
* write a **memory profile** for plotting, see the figure on the right.
* Furthermore, separate `stack_count` function can measure **stack usage**.

The code tool works by intercepting the standard `malloc()`, `free()`, etc
functions. Thus **no changes** are necessary to the inspected source code.

See <http://panthema.net/2013/malloc_count> for the current verison.
Timo Bingmann's avatar
Timo Bingmann committed
15

16
17
18
19
20
21
22
23
24
## Intercepting Heap Allocation Functions ##

The source code of `malloc_count.[ch]` intercepts the standard heap allocation
functions `malloc()`, `free()`, `realloc()` and `calloc()` and adds simple
counting statistics to each call. Thus the program must be relinked for
`malloc_count` to work. Each call to `malloc()` and others is passed on to
lower levels, and the regular `malloc()` is used for heap allocation.

Of course, `malloc_count` can also be used with C++ programs and maybe even
25
26
script languages like Python and Perl, because the `new` operator and most
script interpreters allocations all are based on `malloc`.
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

The tools are usable under Linux and probably also with Cygwin and MinGW, as
they too support the standard Linux dynamic link loading mechanisms.

## Memory Profile and `stack_count`

The `malloc_count` source is accompanied by two further memory analysis tools:
`stack_count` and a C++ header called `memprofile.h`.

In `stack_count.[ch]` two simple functions are provided that can measure the
**maximum stack usage** between two points in a program.

Maybe the most useful application of `malloc_count` is to create a
**memory/heap profile** of a program (while it is running). This profile can
also be created using the well-known
[valgrind tool "massif"](http://valgrind.org/docs/manual/ms-manual.html),
however, massif is really slow. The overhead of `malloc_count` is much smaller,
and using `memprofile.h` a statistic file can be produced, which is directly
usable with Gnuplot.

The source code archive contains two example applications: one which queries
`malloc_count` for current heap usage, and a second which creates the memory
profile in the figure on the right. See the STX B+ Tree library for another,
more complex example of a memory profile.

## Downloads ##

See <http://panthema.net/2013/malloc_count> for the current verison.

The source code is published under the
[MIT License (MIT)](http://opensource.org/licenses/MIT), which is also found in
58
the header of all source files.
Timo Bingmann's avatar
Timo Bingmann committed
59

Timo Bingmann's avatar
Timo Bingmann committed
60
## Short Usage Guide ##
Timo Bingmann's avatar
Timo Bingmann committed
61

Timo Bingmann's avatar
Timo Bingmann committed
62
Compile `malloc_count.c` and link it with your program. The source file
63
64
`malloc_count.o` should be located towards the end of the `.o` file
sequence. You must also add "`-ldl`" to the list of libraries.
Timo Bingmann's avatar
Timo Bingmann committed
65

Timo Bingmann's avatar
Timo Bingmann committed
66
67
Run your program and observe that when terminating, it outputs a line like

Timo Bingmann's avatar
Timo Bingmann committed
68
    malloc_count ### exiting, total: 12,582,912, peak: 4,194,304, current: 0
Timo Bingmann's avatar
Timo Bingmann committed
69
70
71

If desired, increase verbosity

Timo Bingmann's avatar
Timo Bingmann committed
72
1. by setting `log_operations = 1` at the top of `malloc_count.c` and adapting
Timo Bingmann's avatar
Timo Bingmann committed
73
   `log_operations_threshold` to output only large allocations, or
Timo Bingmann's avatar
Timo Bingmann committed
74
2. by including `malloc_count.h` in your program and using the user-functions
75
76
   define therein to output memory usage at specific checkpoints. See the
   directory `test-malloc_count/` in the source code for an example.
Timo Bingmann's avatar
Timo Bingmann committed
77

Timo Bingmann's avatar
Timo Bingmann committed
78
Tip: Set the locale environment variable `LC_NUMERIC=en_GB` or similar to get
Timo Bingmann's avatar
Timo Bingmann committed
79
80
comma-separation of thousands in the printed numbers.

81
82
83
84
85
86
87
88
89
90
91
The directory `test-memprofile/` contains a simple program, which fills a
`std::vector` and `std::set` with integers. The memory usage of these
containers is profiled using the facilities of `memprofile.h`, which are
described verbosely in the source.

## Thread Safety ##

The current statistic methods in `malloc_count.c` are **not thread-safe**.
However, the general mechanism (as described below) is per-se thread-safe. The
only non-safe parts are adding and subtracting from the counters in
`inc_count()` and `dec_count()`.
Timo Bingmann's avatar
Timo Bingmann committed
92

93
94
95
96
97
98
99
100
101
102
103
104
The `malloc_count.c` code contains a `#define THREAD_SAFE_GCC_INTRINSICS`,
which enables use of gcc's intrinsics for atomic counting operations. If you
use gcc, enable this option to make the `malloc_count` tool thread-safe.

The functions in `memprofile.h` are not thread-safe. `stack_count` can also be
used on local thread stacks.

## Technicalities of Intercepting `libc` Function Calls ##

The method used in `malloc_count` to hook the standard heap allocation calls is
to provide a source file exporting the symbols "`malloc`", "`free`", etc. These
override the libc symbols and thus the functions in `malloc_count` are used
Timo Bingmann's avatar
Timo Bingmann committed
105
106
instead.

107
108
109
However, `malloc_count` does not implement a heap allocator. It loads the symbols
"`malloc`", "`free`", etc. directly using the dynamic link loader "`dl`" from the
chain of shared libraries. Calls to the overriding "`malloc`" functions are
Timo Bingmann's avatar
Timo Bingmann committed
110
111
forwarded to the usual libc allocator.

112
To keep track of the size of each allocated memory area, `malloc_count` uses a
113
114
115
116
trick: it prepends each allocation pointer with two additional bookkeeping
variables: the allocation size and a sentinel value. Thus when allocating *n*
bytes, in truth *n + c* bytes are requested from the libc `malloc()` to save the
size (*c* is by default 16, but can be adapted to fix alignment problems). The
117
sentinel only serves as a check that your program has not overwritten the size
Timo Bingmann's avatar
Timo Bingmann committed
118
information.
Timo Bingmann's avatar
Timo Bingmann committed
119
120
121

## Closing Credits ##

122
123
124
The idea for this augmenting interception method is not my own, it was borrowed
from Jeremy Kerr <http://ozlabs.org/~jk/code/>.

125
Written 2013-01-21, 2013-03-16, and 2014-09-10 by Timo Bingmann <tb@panthema.net>