Wrapper scripts for Linux kernel profiling tools, integrated with the crucible performance testing framework.
Kerneltools runs one or more subtools during test execution:
| Subtool | Purpose |
|---|---|
| turbostat | CPU power and frequency monitoring (x86) |
| perf | CPU profiling and hardware event recording |
| intel-speed-select | CPU frequency scaling controls (x86) |
| trace-cmd | Kernel function tracing via ftrace |
| sysfs-trace | Direct kernel trace buffer access via debugfs |
The start script accepts these parameters:
--subtools <list>— Comma-separated subtools to run (default:turbostat)--interval <seconds>— Turbostat collection interval (default:10); does not affect perf bucketing--perf-bucket-ms <ms>— Width of the perf-record CDM time buckets (default:3000)--record-opts <opts>— Additional options passed to perf record--base-freq <freq>— Base frequency for turbostat--turbo-freq <freq>— Turbo frequency for turbostat--core-power— Enable core power reporting in turbostat--trace-cmd-record-opts <opts>— Additional options for trace-cmd record--sysfs-trace-setup <cmd>— Setup commands for sysfs-trace (repeatable)--sysfs-trace-cleanup <cmd>— Cleanup commands for sysfs-trace (repeatable)
The stop script additionally accepts:
--perf-gen-local-report <ON|OFF>— Generate an additional human-readableperf-report.txtalongside the archivedperf.data(default:OFF).kerneltools-stopitself takes this as a no-argument flag;rickshaw.json'sparam_regexrewritesONto the bare flag and dropsOFF(and its value) entirely before the script ever sees it -- mirroring the same ON/OFF-fixup convention used by bench-trafficgen for its own no-argument flags.--sysfs-trace-collection <system|per-cpu>— Trace buffer collection mode (default:per-cpu)
When the perf subtool is enabled, kerneltools-start records the collector's
boot time in kerneltools-boot-time.txt and captures the collector's CPU
topology in kerneltools-cpu-topology.txt, and persists the configured CDM
bucket width in kerneltools-perf-bucket.txt. At stop time, after perf archive and before compressing perf.data, kerneltools-stop generates
perf-report.csv.xz with a perf time quantum matching the configured CDM
bucket width; the controller post-processor aggregates those rows into CDM
buckets perf-bucket-ms milliseconds wide.
Unless --record-opts already contains -e or --event, perf record
collects this default event set:
cycles,instructions,branch-instructions,branch-misses,cache-misses
These five hardware events provide cycle, instruction, branch, and cache-miss
counts while fitting within the non-multiplexed counter budget of the target
hosts. An explicit --record-opts "-e ..." selection replaces the default.
If that set exceeds the available hardware counters, perf multiplexes the
events and reduces their effective sampling rates.
The controller post-processor converts each symbol's sampled event period in each bucket into a CDM sample. The event section in the perf report controls the metric type:
Bucket boundaries stay aligned to the boot-relative timeline: a sample at perf
time t lands in the bucket beginning at floor(t / perf-bucket-ms) * perf-bucket-ms (in milliseconds since boot), so consecutive samples with the
same width produce comparable, non-overlapping buckets. Samples shorter than
the bucket width collapse into a single bucket unless they straddle a bucket
boundary. The structured perf report uses the same configured width as its
time quantum.
When post-processing archived tool data with no kerneltools-perf-bucket.txt
present, the post-processor uses the historical 1000 ms default. A malformed
or non-positive sidecar value falls back to the current 3000 ms default.
cyclesorcpu-cycles(including modifiers such ascycles:u) produces typecpu-cycles.instructions(including modifiers such asinstructions:k) produces typecpu-instructions.branch-instructionsorbranchesproduces typecpu-branch-instructions.branch-missesproduces typecpu-branch-misses.cache-referencesproduces typecpu-cache-references.cache-missesproduces typecpu-cache-misses.- Unsupported events are skipped with a warning so their values cannot be mixed into a metric with different units.
Metric types use the perf event name as the noun and do not append an
aggregation suffix. The count class and sum default aggregation already
describe additive semantics; for example, branch-misses maps to
cpu-branch-misses.
All perf samples use source perf-record, class count, and default
aggregation sum. Breakouts are function, cpu, cmd, package, die,
core, thread, shared-l1-domain, shared-l2-domain,
shared-l3-domain, and shared-l4-domain. The value is the sum of the raw
event Period values for that function, CPU, cmd, event, and time bucket,
avoiding the two-decimal precision loss in perf's displayed Overhead column.
cpu-cycles represents sampled CPU cycles spent in the function on that CPU;
it is not wall-clock or CPU time unless event frequency is known.
cpu-instructions similarly represents sampled retired instructions. Multiple
events, including the default set, produce separate metrics and are never summed
together. The package, die, core, thread, and shared-cache-domain labels
come from the topology captured on the collection host. If a function, CPU, and
cmd combination does not appear in a bucket, no sample is emitted for that
combination in that bucket; gaps are not zero-filled.
Kerneltools runs as a profiler tool on endpoint nodes. It is allowed on profiler, master, and worker collector roles but blocked on client and server roles. Data compression and report generation (perf archive, perf reports, trace-cmd reports) happens at stop time. The CDM post-processor (kerneltools-post-process.py) converts turbostat and perf-record output into crucible's canonical metric format; other subtools are unaffected.