HPCToolkit Tutorial: Master HPC Performance Analysis Step by Step
If you have ever stared at a slow simulation and wondered exactly which function or loop is eating your CPU cycles, an HPCToolkit tutorial is the fastest way to move from guesswork to evidence. HPCToolkit is a mature, open-source suite of performance analysis tools designed specifically for high-performance computing applications, and it has become a standard instrument in the toolbox of developers working on clusters, supercomputers, and even well-equipped workstations. Instead of forcing you to instrument every function by hand, it samples your running program, correlates those samples with source code, and presents the results in a graphical viewer that makes hotspots impossible to ignore.
This guide walks through the complete workflow: installing the toolkit, preparing a code base that can be profiled meaningfully, collecting measurements with hpcrun, converting that raw data with hpcstruct and hpcprof, and interpreting the results in hpcviewer and hpctraceviewer. Along the way you will encounter the practical details that documentation often skips, such as compiler flags that preserve line-level attribution, sampling frequency trade-offs, and how to keep profiling overhead under control on large MPI jobs. By the end, you should be able to run a full measurement and analysis cycle on your own application without needing to consult a second reference.
HPCToolkit Fundamentals and Environment Setup
HPCToolkit is a sampling-based profiler, which means it periodically interrupts your program and records the current call stack rather than inserting counters into your source code. That approach keeps runtime overhead low enough for production-sized runs, while still producing statistically meaningful attribution of time, hardware counters, and memory events to individual functions, loops, and source lines. The suite is built around a small family of command-line tools that each handle one stage of the pipeline, plus two Java-based graphical viewers for interactive exploration.
Because the toolkit relies on binary analysis and on the operating system's ability to unwind call stacks, it works best on Linux systems with standard HPC toolchains such as GCC, Clang, and the Intel or NVIDIA compilers. Support extends across MPI, OpenMP, pthreads, CUDA, HIP, and OpenACC, so hybrid applications are handled without special configuration in most cases. The same measurement directory produced on a compute node can be post-processed on a login node or transferred to a laptop for viewing, which matters when your cluster enforces strict policies about interactive work.
Installing HPCToolkit and Its Dependencies
The least painful installation path on most clusters is through a package manager. Spack ships an HPCToolkit recipe, so spack install hpctoolkit will resolve dependencies such as Boost, elfutils, libunwind, Dyninst, PAPI, and the Python runtime needed for the analysis scripts. Many HPC centers also provide prebuilt modules, in which case a simple module load hpctoolkit followed by hpcrun --version is enough to confirm that the binaries are on your path. If you need a specific version or a custom build, the project's CMake-based build system is well documented and works cleanly once the dependency chain is satisfied.
The graphical viewers deserve separate attention because they are Java applications requiring Qt-based rendering libraries and a working display. On a headless cluster you have three options: run the viewer on a local machine after copying the database over, forward X11 with ssh -X or ssh -Y, or use a remote desktop session. Verifying that hpcviewer launches before you invest time in a long profiling run saves considerable frustration later.
Preparing Your Application for Profiling
Compilation flags determine how useful your profile will be. Build with -g so the compiler emits debug information, keep optimization enabled with -O2 or higher so the measurements reflect realistic performance, and add -fno-omit-frame-pointer to give the unwinder a reliable stack chain on architectures where that matters. Avoid stripping binaries, since HPCToolkit needs symbols and debug sections to map samples back to lines of code.
Selecting an appropriate build configuration is equally important. A profile of a debug build tells you almost nothing about your production workload, and a fully inlined -O3 build can blur the boundary between caller and callee in the call tree. A common compromise is to profile the same optimization level you actually ship, add debug info, and accept that some small functions will be attributed to their inlined parents. Keeping source files accessible at analysis time also helps, because the viewer can then display annotated code alongside the metrics.
Executing the Profiling Workflow
The HPCToolkit pipeline has four stages, and understanding their order prevents most beginner mistakes. First, hpcrun launches your application and collects samples into a measurement directory. Second, hpcstruct analyzes the program binary and reconstructs its loop and call structure. Third, hpcprof merges the measurements, the binary structure, and the source code into a database. Fourth, hpcviewer or hpctraceviewer opens that database for interactive inspection.
Each stage writes to disk and can be rerun independently, which is convenient when you want to re-analyze existing measurements with a different set of metrics or a corrected source tree. Naming your output directories consistently, for example run1.hpcrun and run1.hpcstruct, keeps long profiling campaigns manageable. On multi-node runs, HPCToolkit creates one measurement directory per process, and the analysis tools automatically aggregate them.
Measuring Performance with hpcrun
The simplest invocation is hpcrun -e REALTIME -o myrun ./myapp arg1 arg2, which samples wall-clock time and stores results in a directory prefixed with myrun. Replace REALTIME with counter events such as CPUTIME, PAPI_TOT_CYC, or PAPI_L2_DCM to capture hardware performance counters, and remember that events can be combined on one command line when you want a richer picture. The -t flag enables call-path tracing, which records timestamps and is required if you later want to examine temporal behavior in the trace viewer.
For parallel jobs, prefix the launcher instead of the executable: mpirun -np 64 hpcrun -e REALTIME ./myapp. Keep an eye on overhead, particularly when sampling at high frequencies or when collecting many counters at once. A sampling interval that is too aggressive can perturb the very behavior you are trying to measure, while one that is too sparse may miss short-lived hotspots. Running a short representative workload first, then scaling up once you are confident in your event selection, is the practical way to find a comfortable balance.
Converting Data with hpcstruct and hpcprof
After measurement, run hpcstruct ./myapp to produce a structure file describing the program's loops and functions. This step uses binary analysis and can take several minutes on large executables with heavy template usage, but it only needs to be repeated when the binary changes. For applications built from many shared libraries, point the tool at the relevant binaries or let it discover them automatically from the measurement directory.
The final conversion step is hpcprof -S myapp.hpcstruct -I ./src 'myrun.hpcrun', which combines structure, measurements, and source paths into a database directory. On large jobs, hpcprof-mpi distributes the analysis across multiple ranks and dramatically shortens processing time. Include paths matter here: if the original source directories have moved, supply the new locations so the viewer can resolve file and line information correctly.
Analyzing Results and Tuning Performance
Opening the resulting database in hpcviewer reveals a three-pane layout: a navigation pane holding the call tree, a source pane showing annotated code, and a metrics pane listing the measured values with their percentages of total. The tool is interactive by design, so clicking a node immediately filters the other panes to the corresponding context. Learning a handful of keyboard shortcuts and column sorts turns a large profile into a short list of candidates worth investigating.
For workloads where the time dimension matters, such as phases that alternate between computation and communication, hpctraceviewer offers a timeline view of sampled call paths. Seeing where ranks idle while waiting in a collective operation is often the quickest route to diagnosing load imbalance or an unexpected synchronization point. The two viewers complement each other: one answers "where is the time going," the other answers "when is it going there."
Navigating hpcviewer and the Call Tree
The call tree presents inclusive and exclusive metrics side by side. Inclusive values include everything called beneath a node, while exclusive values cover only the code in that node itself, which is why a function with a large inclusive time but a small exclusive time is usually a coordinator rather than a hotspot. Switch between top-down, bottom-up, and flat views depending on the question you are asking; the flat view is especially effective for spotting the single most expensive function anywhere in the program.
Derived metrics extend the analysis further. You can compute ratios such as cycles per instruction or cache misses per thousand instructions directly in the viewer, then sort source lines by that ratio to expose code that is slow per unit of work rather than merely frequently executed. Filtering by depth, by MPI rank, or by thread helps isolate behavior that only appears under specific conditions.
From Metrics to Optimization Decisions
Interpreting a profile is a matter of matching evidence to remedy. A loop with high exclusive time and a low instructions-per-cycle value suggests poor vectorization or excessive branching, so inspect the compiler's vectorization report and consider restructuring the inner loop. High last-level cache miss rates point toward blocking, data layout changes, or reordering of array traversals. Large idle time distributed unevenly across ranks signals a load balancing problem that may require dynamic scheduling or a different decomposition.
Profiling is most effective as a loop rather than a one-time audit: change one thing, rebuild with the same flags, run the same workload, and compare the new database against the old one using the viewer's comparison feature. Keeping a short written log of measurement conditions prevents the classic mistake of attributing an improvement to a code change that was actually caused by a different input size or node count.
With a working measurement pipeline in place, the natural next step is to profile something you actually care about and let the data choose your optimization targets. Start with a small, representative run, validate that the call tree matches your mental model of the program, and only then scale up to full production sizes. The habits you build during that first careful cycle will pay off every time a new performance mystery lands on your desk.