Below is a list of Linux CLI tools for profiling AI inference engines on Linux. I use these for optimising Kernels.
Kernel-Level Profiling
| Command |
Description |
Install |
rocprof |
Legacy kernel profiler. Per-kernel dispatch counts, durations, and hardware counters. Use --stats for aggregate tables, --metrics for HW counters (occupancy, waves, LDS pressure). |
pre-installed with ROCm |
rocprofv3 |
New-generation profiler (replaces rocprof). Supports system tracing (--sys-trace), kernel dispatches (--kernel-trace), memory copies (--memory-copy-trace), and outputs Perfetto format (-f pftrace). Lower overhead, better HIP graph attribution. |
sudo dnf install rocprofiler-sdk |
rocprofv3-avail |
Lists available counters, agents, and metrics for rocprofv3. |
sudo dnf install rocprofiler-sdk |
rocprofv3-attach |
Attaches rocprofv3 to an already-running process. |
sudo dnf install rocprofiler-sdk |
Timeline & System Tracing
| Command |
Description |
Install |
perfetto |
Trace collection daemon and viewer. Opens .pftrace files from rocprofv3 for timeline visualisation (GPU kernel gaps, cross-GPU skew, API call latency). Run perfetto -o trace.pftrace -t 10s for system-wide tracing. |
sudo dnf install perfetto |
rocprof-sys-run |
Runs a binary under rocprofiler-systems (runtime instrumentation + binary rewriting). Captures function-level hotspots without recompilation. |
sudo dnf install rocprofiler-systems |
rocprof-sys-sample |
Sampling profiler — attaches to a running process and collects statistical samples. Low-overhead alternative to full tracing. |
sudo dnf install rocprofiler-systems |
rocprof-sys-causal |
Causal profiling — measures the impact of hypothetical speedups by slowing down other parts. Answers "if I fix X, how much does total improve?" |
sudo dnf install rocprofiler-systems |
rocprof-sys-instrument |
Instruments a binary with rocprofiler-systems markers without running it. |
sudo dnf install rocprofiler-systems |
rocprof-sys-avail |
Lists available probes and instrumentation points. |
sudo dnf install rocprofiler-systems |
GPU Tracing Library (link-time)
| Command |
Description |
Install |
libroctracer64.so |
Link-time tracing library (-lroctracer64). Callback API for HIP runtime calls, kernel dispatches, and memory operations with nanosecond timestamps. Embed tracing into your own tool. |
sudo dnf install roctracer-devel |
libroctx64.so |
User marker library (-lroctx64). Add named ranges (roctxRangePush("layer 3")) that appear in rocprof/rocprofv3 traces for phase attribution. |
sudo dnf install roctracer-devel |
Debugging
| Command |
Description |
Install |
rocgdb |
ROCm-aware GDB (16.3). Debugs GPU kernels with breakpoints, variable inspection on device memory, and multi-GPU thread control. Use rocgdb --args ./binary like regular gdb. |
pre-installed with ROCm |
Monitoring (Python)
| Library |
Description |
Install |
amdsmi |
Python library (26.2.1) for GPU metrics: per-CU utilisation, memory bandwidth, clock frequencies, PCIe throughput, temperature, power. import amdsmi; amdsmi.amdsmi_init() |
pip install amdsmi |
Quick Reference
| Task |
Command |
| Per-kernel stats |
rocprof --stats ./binary |
| Full system trace |
rocprofv3 --sys-trace -f pftrace -- ./binary |
| View trace timeline |
open .pftrace at ui.perfetto.dev or perfetto locally |
| Hardware counters (occupancy, LDS, waves) |
rocprofv3 -i counters.txt -- ./binary |
| Debug GPU crash |
rocgdb --args ./binary |
| Sampling profile (low overhead) |
rocprof-sys-sample -p <pid> |
| Causal analysis ("what if X were faster?") |
rocprof-sys-causal ./binary |
| GPU utilisation / VRAM |
rocm-smi or python3 -c "import amdsmi" |