Survey of Software Monitoring and Profiling Tools

Ben Wun [bw6@cse.wustl.edu]

Abstract

Table of Contents:

• 1. Introduction
• 2. Static Program Modification
  • 2.1 Gprof
  • 2.2 ATOM
  • 2.3 Etch
  • 2.4 Summary
• 3. Hardware Counters
  • 3.1 Oprofile
  • 3.2 MIPS R10000
  • 3.3 Cell Processor
  • 3.4 Summary
• 4. Kernel Profiling
  • 4.1 DTrace
  • 4.2 Other Kernel Profilers
  • 4.3 Summary
• 5. Virtual Machines
  • 5.1 XenoProf
  • 5.2 Summary
• 6. Parallel Systems
  • 6.1 ZM4/SIMPLE
  • 6.2 Summary
• 7. Summary
• 8. References
• List of Acronyms

1. Introduction

The characteristics of a good program monitor can differ depending on the exact scenario being measured. It may be adequate to profile a single program under study, or it may be more desirable to measure whole system performance. However, there are several goals that are shared by all monitoring systems. First and foremost, the program behavior should be modified as little as possible. If a program's behavior is radically modified by monitoring it, then the collected data may be worthless. Furthermore, if the instrumentation is an integral part of the system's design, it should not impose any overhead when inactive. For example, if hardware counters are included in a processor, they should not lie on the critical path of the circuit's logic. Flexibility is another attribute of a good system monitor. It is possible for a monitoring program to collect too little, too much, or the wrong kind of data. For example, if a program is hampered by cache misses, a call graph profiler may be of little use. On the other hand, it is possible to collect so much data that it becomes difficult if not impossible to analyze it. Good system monitoring tools allow the user to specify the amount and type of data to be collected.

This paper organizes the tools to be discussed as follows- section 2 discusses static program modification, section 3 discusses hardware support for performance profiling, section 4 discusses kernel profiling, section 5 discusses monitoring of virtual machines, and section 6 discusses monitoring of parallel systems. We summarize this information in section 7.

Back to Table of Contents

2. Static Program Modification

There are several points at which a program may be instrumented. The first is at compile time. If the source code of the program and the compiler are available, the compiler can be modified to instrument the program. This has the advantage of the availability of symbolic and syntactical information that is not available to binary rewriters. However, access to the source code for both the compiler and program is not always possible. It is also not possible to instrument any libraries that may be linked to later. Using the linker to insert checks can resolve this problem.

If the only thing that is accessible is the program binary, the above methods will not work. The other option is to use an executable rewriter that instruments the binary executable directly. This has the advantage of working for any program, not just those for which the source code is available. However, binary rewriters may not have access to all the symbolic debugging information.

In this section, we will examine three examples of program profilers that use static program modification. Each instruments the program at a different stage, and table 1 summarizes their characteristics.

2.1 Gprof

Gprof relies on having the symbolic information available in source code in order to properly construct the call graph. Any libraries that are linked in must be compiled with support for gprof, or profiling data cannot be gathered from them. Furthermore, only certain languages are supported. When a program is compiled with gprof support, monitoring routines are inserted at strategic points in the code to produce a trace of events. As little work as possible is performed at runtime in order to avoid adversely affecting program behavior. More extensive analysis of this trace is performed off line in order to recreate the call graph. Since Gprof is implemented at the language level, it is portable across different architectures and OSes.

Gprof is very easy to use and portable, but it is limited in scope. It is only designed to produce a call graph and to tabulate the time spent in each function. Furthermore, access to the source code is required to use it. A more flexible framework that allows users to define how they want to instrument their programs is discussed next.

2.2 ATOM

When using ATOM, a programmer must write instrumentation routines that tell ATOM where to insert code, and what type of code should be inserted. Users write instrumentation routines to tell ATOM where to insert instrumentation code, and analysis procedures that define what should happen at those instrumentation points. An example instrumentation routine that records the effective address at each load follows:

Instrument(int iargc, char**iargv)
{
Proc* p;
Block *b;
Inst *inst;

AddCallProto("OpenRecord()");
AddCallProto("CloseRecord()");
AddCallProto("Load( VALUE)");

for(p=GetFirstProc();p!=NULL;p=GetNextProc(p)){
    for(b=GetFirstBlock(p);b!=NULL;b=GetNextBlock(b)){
       for (inst = GetFirstInst(block); inst != NULL; inst = GetNextInst(inst)){
           if(IsInstType(inst, InstTypeLoad){
              AddCallInst(inst, InstBefore, "Load", EffAddrValue);
          }
       }
    }
}
AddCallProgram(ProgramBefore, "OpenRecord");
AddCallProgram(ProgramAfter, "CloseRecord");
}

ATOM breaks the program up into procedures, and the procedures into basic blocks. The basic blocks are further broken down into individual instructions. The GetNextProc and GetNextBlock functions return a handle to the next procedure or basic block respectively. The GetLastInst and IsInstType routines operate on individual instructions within the current basic block. The AddCallProto primitive defines function prototypes for analysis procedures defined elsewhere. In this example Load, OpenRecord and CloseRecord are the analysis routines. The Load routine is passed the Effective address used for the load, which will be of use to the analysis routine. AddCallInst adds a call to the indicated analysis routine at the specified point. Next, we define the analysis routines:

File* f;
void OpenRecord()
{
    f = fopen("LoadAddrs.txt");
}

void Load(int addr)
{
    fprintf(f, "%X\n", addr);
}

void CloseRecord()
{
   fclose(f);
}

The open and close record routines are self explanatory. The Load routine takes as an argument the address that is being loaded and records it in the file.

ATOM imposes what its creators consider to be acceptable overhead. Using the SPEC92 benchmark suite, they found that ATOM increased execution times anywhere from 1.01x to 11.84x of the uninstrumented code.

While ATOM's use of object files as inputs frees it from dependence on any particular language or compiler, access to the intermediate object files is still needed. If a user wants to observe the behavior of a program for which only the binary executable is available, another tool, such as the one discussed in the next section, is needed.

2.3 Etch

2.4 Summary

Static instrumentation of individual binaries can be a powerful tool for observing the runtime behavior of a program. The main drawback to this approach is that it is limited in scope; the other layers of the system, such as the operating system and hardware, can have important effects on the execution of a program. Events such as scheduling, cache misses, interrupt handling and more can be the underlying cause of poor system performance, but these are not captured by the methods discussed in this section, and can only be obtained with the help of code in the OS itself, or with hardware monitoring facilities.

Back to Table of Contents

3. Hardware Counters

3.1 Oprofile

Oprofile is the Linux kernel's framework for accessing hardware counters. Others, such as Vtune[Vtune] exist, but will not be discussed because they operate on similar principles. For more details about the counters themselves, we will now take a look at their implementation in one particular processor, the MIPS R10000.

3.2 MIPS R10000

Hardware designers must be careful when incorporating hardware counters in their chips. The additional logic must not be prohibitively expensive, and must not lie on the critical path, or would degrade performance, even when they are not in use. The performance counters for the MIPS R10000[Zagha96] use 300 lines of RTL and fewer than 5000 transistors. This was less than 0.2% of the number of transistors used in the entire processor, not counting cache. The design team felt that "the counters were well worth the silicon and design time to include them."

Now that we have discussed how hardware counters are integrated into the processor and OS, we will look at an example of their use. The authors of the MIPSR10000 paper [Zagha96] used their counters to examine the the effects of multiprocessor sharing in a system that uses a cache-snooping protocol. The authors examined a weather modeling program that did not scale well to multiple processors. The underlying cause was puzzling, because threads in this program share very little data and should run fairly independently. Using the hardware counters, the authors could determine the number of cycles the program took to execute, as well as the number of cache invalidation events and the cycles needed to service them. This data indicated that execution time was dominated by secondary cache misses, caused by a large number of invalidations. This would not have been obvious by examining the source code, because very little sharing occurs between threads in this program. It was found that false sharing, caused by allocating non shared variables to the same cache line, was the cause of the problem.

The model of programmable counters that interrupt the processor when they overflow, is the most common one, but it imposes an overhead on program execution that may have adverse effects on the system being studied. A newer generation of processors uses more advanced facilities to try to overcome this problem, as we discuss in the next section.

3.3 Cell Processor

3.4 Summary

4. Kernel Profiling

4.1 DTrace

Dtrace defines several thousand points in the kernel that can be instrumented. A user writes providers, which are loadable kernel modules that use the DTrace API to communicate with the kernel. The providers create probes at the instrumentation points of interest. The probes are then provided to user space consumers, which are responsible for enabling the probes and aggregating the data. When the probe is fired, the DTrace framework preempts the system and takes over. In order to improve performance and cut down on the amount of data that the system must deal with, probes can be associated with a predicate that lays out the conditions under which the probe will fire. If the conditions are not met, the probe will not be invoked. Another mechanism to address this is the use of speculative tracing. It may not be apparent that data can be discarded until after the probe has fired, and speculative tracing allows this data to be buffered and discarded at a later time if deemed appropriate.

In order to both improve performance and reduce the amount of data that needs to be processed, DTrace allows the user to specify predicates, which determine if a particular enabled probe is to be run. Probes will only run if the conditions of the predicate are met. If the predicate is met, the probe fires and executes a user defined action. Users specify this action in the D language, which is compiled into the D Intermediate Format (DIF), which is then interpreted by a virtual machine in the kernel. Several provisions, like the elimination of non-forward branches, and runtime safety checks, ensure that these user probes are safe to run in kernel mode. This is a very powerful idea, since it allows users to define how their probes should behave without having to worry about bringing down a running system.

Instrumentation of user programs is achieved through what its creators call a pid provider. As with kernel probes, there is zero disabled probe effect for user space programs, and the user program need not be restarted to be instrumented. Instead of rewriting the program to branch to instrumentation code, DTrace rewrites the target instruction to induce a kernel trap. This allows for a unified kernel and user space monitoring mechanism at the cost of runtime overhead.

DTrace aims to be a comprehensive performance evaluation and monitoring framework, but one major feature that it lacks is support for hardware counters. Furthermore, it is not portable across different OSes.

4.2 Other Kernel Profilers

4.3 Summary

The OS is not always the lowest layer in a system's software stack. Virtual machine technologies allow multiple OSes to share a single underlying hardware system. This has implications for performance monitoring that we will discuss in the next section.

Back to Table of Contents

5. Virtual Machines

5.1 XenoProf

As an example of the utility of whole system monitoring in a virtual machine environment, the XenoProf creators used their creation to profile the network subsystem of a Xen system. They found that the virtualized system performed far worse than the same hardware without virtualization. Using XenoProf, they were able to trace the cause to the high cost of page remappings in the virtualized network driver. Armed with this data, they were able to greatly improve the performance of the Xen virtualized network driver.

5.2 Summary

All of the techniques discussed up to this point were built to monitor a single computer system. When a parallel computing cluster consisting of multiple nodes is being studied, new tools must be developed to account for the special challenges in such an environment. As we will see in the next section however, the mechanisms used in single node systems are still useful.

Back to Table of Contents

6. Parallel Systems

6.1 ZM4/SIMPLE

ZM4[Hofmann94] is the hardware component of the ZM4/SIMPLE system. ZM4 consists of a control and evaluation computer (CEC) that controls the actions of multiple, distributed monitor agents (MA). MAs are PCs equipped with 4 dedicated probe units (DPUs). DPUs are custom printed circuit boards which link the MAs to the systems they monitor. DPUs perform tasks such as time stamping, event recording and buffering of said events. The local clock of the MA has a resolution of 100ns (which was considered adequate when the paper was published) which is synchronized to a global measure tick generator(MTG). In order for this to work, the wire length between each node and the MTG must be known. Each MA can be up to 1000m away from the CEC. Ethernet is used for communication between the MAs and CEC. In addition to the ZM4 monitor, the software is also instrumented at key points. This is necessary in order to correlate the hardware events to the software's symbols and structure.

Inferring the order of events on distributed nodes requires special consideration in a massively parallel system. A global ordering must be derived from local events. In a message passing system, there is a global ordering inherent in the send and receive operations. In a shared memory system, the use of synchronized global clocks, as provided by ZM4, is necessary to decide which accesses to shared addresses occur in what order. The timer must have resolution less than memory access times to be successful.

6.2 Summary

Back to Table of Contents

7. Summary

Back to Table of Contents

8. References

1. [Burkhart89] H. Burkhart, R. Millen. "Performance-measurement tools in a multiprocessor environment." IEEE Transactions on Computers, Vol. 38, No. 5, May 1989.
   http://ieeexplore.ieee.org/document/24274 /

2. [Cantrill04] Bryan M. Cantrill, Michael Shapiro, Adam Leventhal. "Dynamic Instrumentation of Production Systems." In Proceedings of the General Track: 2004 USENIX Annual Technical Conference, June 2004.
   http://www.usenix.org/event/usenix04/tech/general/full_papers/cantrill/cantrill.pdf

3. [Genden06] Mike Genden, Ram Raghavan, Mack Riley, John Spannaus, Thomas Chen, "Real-time Performance Monitoring and Debug Features of the First Generation Cell Processor," In Proc. of 1st Workshop on Tools and Compilers for Hardware Acceleration, September 2006

4. [Graham82] Susan L. Graham, Peter Kessler, Marshall Mckusick. "Gprof: A call graph execution profiler." ACM Sigplan Noticies, June 1982.
   http://portal.acm.org/citation.cfm?id=872726.806987&coll=portal&dl=ACM&CFID=11111111&CFTOKEN=2222222

5. [Hofmann94] R. Hofmann, R. Klar, B. Mohr, A. Quick, M. Siegle. "Distributed Performance Monitoring: Methods, Tools and Applications." IEEE Transactions on Parallel and Distributed Systems, Vol 5, No. 6, June 1994.
   http://csdl2.computer.org/persagen/DLAbsToc.jsp?resourcePath=/dl/trans/td/&toc=comp/trans/td/1994/06/l6toc.xml&DOI=10.1109/71.285605

6. [Larus94] James R. Larus and Thomas Ball. "Rewriting executable files to measure program behavior." Software, Practice and Experience, vol 24, no. 2, pp. 197-218, Feb 1994. http://research.microsoft.com/~tball/papers/rewrite.pdf

7. [Menon05] Aravind Menon et. al. "Diagnosing Performance Overheads in the Xen Virtual Machine Environment." Proceedings of the 1st ACM/USENIX international conference on Virtual execution environments, 2005.
   http://portal.acm.org/citation.cfm?id=1064979.1064984

8. [Moore01] Richard J. Moore. A universal dynamic trace for Linux and other operating systems. In Proceedings of the FREENIX Track, June 2001. https://www.usenix.org/events/usenix01/freenix01/moore.html

9. [Oprofile] Oprofile – A System Profiler for Linux: http://Oprofile.sourceforge.net/news/

10. [Ries93] B. Ries, et. al. "The paragon performance monitoring environment." Proceedings of the 1993 ACM/IEEE Conference on Supercomputing, pp. 850-859, 1993.
    http://portal.acm.org/citation.cfm?id=169627.169851

11. [Romer97] Ted Romer et. al. "Instrumentation and Optimization of Win32/Intel Executables Using Etch." Proceedings of the USENIX Windows NT Workshop, August 1997.
    http://www.usenix.org/publications/library/proceedings/usenix-nt97/full_papers/romer/romer.pdf

12. [Srivastava94] Amitabh Srivastava, Alan Eustace. "ATOM: a system for building customized program analysis tools." Proceedings of the ACM SIGPLAN 1994 conference on Programming language design and implementation, 1994.
    http://portal.acm.org/citation.cfm?id=178260&coll=portal&dl=ACM&CFID=11111111&CFTOKEN=2222222&ret=1#Fulltext

13. [Vtune] Intel VTune Performance Analyzers. www.intel.com/vtune

14. [Yaghmour00] Karim Yaghmour and Michel R. Dagenais. Measuring and characterizing system behavior using kernel-level event logging. In Proceedings of the 2000 USENIX Annual Technical Conference, 2000.
    http://www.usenix.org/publications/library/proceedings/usenix2000/general/full_papers/yaghmour/yaghmour_html/

15. [Zagha96] M. Zagha, B. Larson, S. Turner, M. Itzkowitz. "Performance Analysis Using the MIPS R10000 Performance Counters." 1996 ACM/IEEE Conference on Supercomputing, 1996.
    http://ieeexplore.ieee.org/document/1392891/

List of Acronyms

Back to Table of Contents