Repository navigation
Conversation
…cks up intel#33) (intel#38) * Add dtoctl wrapper to drive DTO without manual LD_PRELOAD setup Using the DTO library today requires the user to understand two orthogonal mechanisms before a single program can be offloaded to DSA. The library has to be injected into the target through LD_PRELOAD, and its behavior is steered entirely through a set of roughly twenty DTO_* environment variables that must be discovered by reading the manual and then exported by hand. This is easy to get wrong: the LD_PRELOAD path and the variable names are stringly typed, an exported variable silently leaks into every later command in the same shell, and there is no validation or discoverability at the point of use. Provide a small wrapper, dtoctl, that turns this into a single self-describing command line. It follows the same model as hmctl from the hmsdk project: parse options, translate them into the environment the library expects, and then exec the target program so no state lingers in the caller's shell. On top of that model dtoctl also takes care of the injection itself by prepending the resolved libdto.so to LD_PRELOAD, which is possible because DTO works by transparently interposing on the standard memory APIs. Every DTO_* variable is exposed as an option so the mapping stays obvious and greppable. Only options that are actually given are written to the child environment; anything left unset keeps whatever the caller already exported or the library's own default, and an explicit option wins over an inherited variable. The library path is resolved from -l/--library, then $DTO_LIBRARY, then a compiled-in default, and execvp() is used so a bare command name on PATH works as well as an explicit path. Build and install dtoctl with both Make and CMake. Set its compiled-in library path to match the installation directory, since CMake's default /usr/local/lib differs from the Makefile's /usr/lib64. Allow that path to be overridden with DTOCTL_LIBPATH. With this in place the previous incantation $ export LD_PRELOAD=/usr/lib64/libdto.so $ export DTO_WAIT_METHOD=busypoll $ export DTO_CPU_SIZE_FRACTION=0.33 $ ./prog collapses to $ dtoctl -w busypoll -c 0.33 ./prog Signed-off-by: Honggyu Kim <honggyu.kim@sk.com> * Add a dtoctl man page The dtoctl options mirror the DTO_* environment variables, but a user coming to the tool for the first time still needs a single authoritative place that explains what each option does, what values it accepts, and how the library path and LD_PRELOAD are resolved. Scattering that information across the source help text alone makes it hard to consult without running the tool. Add a manual page in the same troff-friendly markdown form the hmsdk project uses for hmctl, and render it as a section 1 man page for the user command. Besides the per-option reference it documents the library resolution order and the override semantics, and it gathers the common tuning goals - latency, power, and cache-pollution avoidance - as ready-to-run examples such as $ dtoctl -w umwait -c 0.0 --no-auto-adjust ./prog so users can map an intent directly onto a command instead of reverse engineering it from the individual variables. Keep the markdown as the source form and generate dtoctl.1 with pandoc during the Makefile and CMake builds. Both build systems install the generated page under man1, so it can be read with $ man 1 dtoctl Pandoc is optional: when it is unavailable, print a message recommending its installation and skip manual generation. The Makefile can still install an already generated page. Ignore its generated doc/dtoctl.1 and remove it on make clean; CMake keeps its generated page in the build directory. Signed-off-by: Honggyu Kim <honggyu.kim@sk.com> * Document dtoctl in the README The README is where users first learn how to configure and run DTO, and until now it only described the manual LD_PRELOAD and DTO_* export workflow. A reader following it has no reason to know that dtoctl exists or that it removes most of that ceremony, so the easier path stays hidden precisely where people look for guidance. Introduce dtoctl after the environment variable reference and the common usage models that accompany it, so that discussion stays in one piece and the new section can point back at those usage models. Contrast the old export sequence with the equivalent $ dtoctl -w busypoll -c 0.33 ./prog and spell out the resolution order for libdto.so along with the rule that only the options actually passed are applied. Keep the per-option reference in dtoctl(1) instead of repeating it here. The option list already exists in the --help output and in the manual, and a third copy in the README would only be one more place to forget when an option changes. Signed-off-by: Honggyu Kim <honggyu.kim@sk.com> * dto: honor DTO_WAIT_METHOD=yield The yield wait method is implemented in dsa_wait() and documented in the README, but dsa_init() never matched "yield" when parsing DTO_WAIT_METHOD, so DTO_WAIT_METHOD=yield silently ran with busypoll. * dtoctl: validate options, resolve the library up front, add --dry-run Make the wrapper catch the mistakes it exists to prevent instead of passing them on to libdto, which silently falls back to a default: - Validate every option value: wait method names, integer ranges for --numa-aware and --log-level, a cpu fraction in [0, 1), a positive umwait delay, and non-empty paths and WQ list. --min-bytes takes an optional K/M/G suffix and is passed on as plain bytes; libdto would read "64K" as 64 and "abc" as 0. - Resolve libdto.so to an absolute path and fail with exit status 125 if it cannot be read. ld.so only warns about a missing preload and runs the program without DTO, and a relative path stopped working as soon as the program changed directory. When built with LIBDTO_REL_PATH, prefer the library installed next to dtoctl so a relocated install finds its own copy. - Drop other libdto.so entries from an inherited LD_PRELOAD, so one process never loads two copies of DTO. - --stats-file implies --stats; libdto ignores DTO_STATS_FILE otherwise. - Add -i/--ignore-env to unset inherited DTO_* variables. The --no-* flags can only turn features off, so this is the way to undo an exported DTO_USESTDC_CALLS=1 or DTO_DSA_MEMCPY=0. - Add --dry-run to print the resolved library, LD_PRELOAD, each DTO_* variable with whether it came from an option or the environment, and the command, without running it. - Add --version, and exit with 126/127 when the program cannot be run or is not found, as env(1) and timeout(1) do. - Make the options local to main() and drop dead argument handling. * dtoctl: follow the install prefix, report the version, add tests CMake cached the default library path on the first configure, so a later CMAKE_INSTALL_PREFIX change was ignored, and "cmake --install --prefix" produced a dtoctl that preloaded /usr/local/lib/libdto.so. Compute the default on every configure, pass dtoctl the library's path relative to its own bindir so a relocated install finds its own copy, and keep DTOCTL_LIBPATH as an explicit override. Pass the project version to both builds for --version, and make the Makefile's install target build dtoctl first. Add tests/test_dtoctl.sh as the "dtoctl" CTest (label functional). It needs no DSA: it checks the option to variable mapping, validation, library resolution, LD_PRELOAD handling and exit statuses through --dry-run and the program's environment, and that libdto applies the settings through its own configuration trace. * docs: document dtoctl's validation, library resolution and --dry-run Update the dtoctl man page and README section for the new behavior: option validation, -i/--ignore-env, --dry-run, --version, size suffixes, --stats-file implying --stats, how the library is resolved, and the exit statuses. Add a note on sudo, which resets the environment and strips LD_PRELOAD when it sits between dtoctl and the program, and on settings being inherited by child processes. Fix stale defaults in the README's variable list: DTO_WAIT_METHOD defaults to busypoll and DTO_MIN_BYTES to 65536, tpause is a valid wait method, and DTO_LOG_FILE gets .<program>.<pid> appended. Describe the dtoctl test in TESTING.md. * gitignore: ignore the libdto_explicit build output --------- Signed-off-by: Honggyu Kim <honggyu.kim@sk.com> Co-authored-by: Honggyu Kim <honggyu.kim@sk.com>
test_perf.c uses fork(), _exit(), sysconf() and _SC_PAGESIZE without including <unistd.h>. With GCC 12 the transitive include is missing and the CMake build fails with "_SC_PAGESIZE undeclared".
…ntel#43) With DTO_WAIT_METHOD=tpause, init_dto() converts the wait from ns to TSC cycles using CPUID leaf 0x15 and divides by the denominator unconditionally. If the leaf is absent (__get_cpuid() returns 0 with the outputs uninitialized) or reports den=0, as on many VMs, the constructor dies with SIGFPE before main(). It also ran after dsa_init() had failed and the library was falling back to CPU, where the value is unused. Only do the conversion when DSA is in use and tpause is selected, and require the leaf to be present with non-zero den, num and frequency. Otherwise log an error and fall back to busypoll. Also fix waitpkg detection in dsa_init(), which queried CPUID leaf 0 instead of leaf 7. Leaf 0's ECX is the ASCII "ntel", which has bit 5 set, so waitpkg_support was always 1 and tpause/umwait on a CPU without WAITPKG would SIGILL instead of falling back.
…ntel#40) get_wq() picked a work queue with next_wq++ on a global atomic (and on a per-device atomic in NUMA-aware mode), so every DSA submit on every thread did a locked read-modify-write on one shared cache line. Each thread now keeps its own round-robin position. A thread takes its starting position once from a shared counter, so concurrent threads still spread across the queues, and from then on picking a queue touches only thread-local state. The position is 32 bits, so the rotation no longer skews each time the old atomic_uchar counter wrapped at 256 with a queue count that does not divide it. Measured on a Xeon 6787P with four DSA devices on one node, one shared WQ each: N pinned threads each doing back-to-back memcpy of one size through LD_PRELOAD, auto-tuning off, ns per call (median of 3): 4 KiB 16 threads 2295 -> 1432 32 threads 6629 -> 1549 16 KiB 32 threads 5920 -> 2991 64 KiB 32 threads 16970 -> 12124 Single-thread and device-bound cases are unchanged within noise, and the descriptors stay evenly spread: DSA perfmon counted the same number on each of the four devices in every NUMA mode, before and after. Claude-Session: https://claude.ai/code/session_01Mv7xKtQPkGoXQUvvuuhAEz Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
When another project pulls DTO in with add_subdirectory or FetchContent: - The perf test's build steps used CMAKE_SOURCE_DIR and CMAKE_BINARY_DIR, which are the parent's, so tests/build_baseline.sh and dto.c were not found and the parent's build failed -- with DTO's default options. - DTO_BUILD_TESTS defaulted to ON, adding DTO's tests, its dto-test apps and a perf baseline fetched from GitHub to the parent's build. - An unset CMAKE_BUILD_TYPE was forced to RelWithDebInfo in the cache, changing the parent's build type. Use the CMAKE_CURRENT_* directories, default DTO_BUILD_TESTS and the new DTO_BUILD_TEST_APPS (dto-test, dto-test-wodto) to ON only when DTO is the top-level project, and leave the build type alone otherwise. A top-level build is unchanged.
DTO_COLLECT_STATS kept one global table that every thread updated with
atomic adds, so the cost of collecting grew with the thread count: a
4 KiB memcpy on the CPU path cost 55 ns more with stats on at one
thread and 2.6 us more at 32 threads. Each thread now counts in its own
block, mapped when the thread first records a call and folded into a
retired total when the thread exits; output sums the live blocks and
the retired total. Stats now cost about 40 ns per call (the two clock
reads) at any thread count:
extra cost per call, CPU path 1 thr 4 thr 16 thr 32 thr
4 KiB, before +55 +357 +1319 +2612 ns
4 KiB, after +44 +46 +42 +37 ns
The histogram's 512 linear 4 KiB buckets, the last of which held
everything from 2 MiB up, become log-scaled: under 64 bytes, then four
buckets per power of two up to 4 GiB, then one for anything larger.
New settings:
DTO_STATS_INTERVAL_MS=N a sampler thread writes what happened in
each N ms; the records add up to the summary
DTO_STATS_FORMAT=json JSON Lines: interval and summary records with
calls/bytes/ns per op and path, failures,
histogram, and the configuration at exit
DTO_STATS_FILE=...%p... %p becomes the process ID
DTO_STATS_FILE is now appended to rather than truncated, so a forked
or exec'd child's records don't overwrite its parent's; each record
carries its pid. After fork the child starts with empty counts, and a
%p file is reopened under its own pid. The old dto_stats_log() wrote
each line twice when DTO_LOG_FILE was set and reused a consumed
va_list; it is gone, along with the TRACE log level that turning on
stats forced.
Adds tests/test_stats.py (ctest -R stats) with a workload of known
calls: totals and buckets, intervals summing to totals, 200
short-lived threads all counted and their blocks unmapped, fork,
%p, and text output.
The stats rework uses thread keys, a sampler thread and its condition variable. Before glibc 2.34 those functions are in libpthread, not libc, so preloading libdto into a program that doesn't link libpthread itself (sh, true, most single-threaded tools) failed at load time: true: symbol lookup error: libdto.so: undefined symbol: pthread_key_create Link libdto with pthread in CMake and the Makefile (an empty stub on newer glibc), and check in the stats test that stats work in a program without libpthread. Seen on RHEL 8 (glibc 2.28); Ubuntu 24.04 (2.39) is not affected.
- stats_fork_child() called pthread_setspecific(stats_key, NULL) even when stats were off and the key had never been created, i.e. on key 0, which belongs to whichever library created it first. Only touch the key when it exists. - sampler_lock was not held across fork(), so a child could inherit it locked by the parent's sampler thread and hang in its first stats_start_sampler(). Take it in the prepare handler with the other two and release it in parent and child. - stats_stop_sampler() reset sampler_started, so a thread whose first counted call landed after cleanup_dto() restarted the sampler during process exit, writing to closed descriptors. Set stats_shutdown in cleanup_dto() and check it before starting the sampler. - stats_flush() gave up on EINTR and EAGAIN and discarded the rest of the record. Retry on EINTR and poll() on EAGAIN. - The 64 KiB output buffer was smaller than a worst-case JSON summary, so a record could go out in two write()s and interleave with another process's in a shared O_APPEND file. Make it 256 KiB. - DTO_STATS_INTERVAL_MS accepted negative, non-numeric and absurd values silently; reject them with an error like DTO_STATS_FORMAT does.
"dtoctl --stats-run[=DIR] [--repeat=N] [--perf[=PERF]] prog" runs the program CPU-only (DTO_USESTDC_CALLS=1) and then with DSA, N times each, alternating, with DTO_STATS_FORMAT=json and each run's stats in its own file in DIR. dtoctl records each run's wall time, user and system CPU time (wait4, so including children) and exit status in DIR/runs.jsonl, then runs dto-stats-report on DIR. dto-stats-report (Python 3, standard library only) sums the stats of every process a run started, takes medians over the runs, and prints: wall, CPU and mem* time each way; Amdahl's law from f (share of the CPU-only run's CPU time in mem* calls) and s (mem* speedup per byte), 1 / ((1 - f) + f / s), against the measured speedup and the 1 / (1 - f) limit; each operation and the sizes with the most time; and notes: no call reached DSA (and why), runs did different amounts of work, sizes where DSA was slower (with a DTO_MIN_BYTES to try), runs that failed or wrote no stats. --json prints the numbers instead. With --perf, one more run each way goes through "PERF record -e cpu-clock" (cpu-clock works without hardware counters, and on a test kernel here cycles hardly sampled). perf runs dtoctl again, so DTO is preloaded into the program and not into perf. The report splits each profile into copies, memset, memcmp, CRC, kernel copies, DTO itself and the rest, and notes mem* time DTO did not intercept and CRC time DSA could take. Where libc has no symbol table (Ubuntu), perf shows its memcpy as an address; on the machine that made the profile the report asks its own libc, through dlsym, which implementation of each function this CPU uses, and places samples by address, after checking the profile's libc has the same build ID. Ctrl-C and Ctrl-\ go to the program, as with system(); a run that fails stops the rest, and dtoctl exits with its status. Also adds --stats-format and --stats-interval for DTO_STATS_FORMAT and DTO_STATS_INTERVAL_MS, and documents them with --stats-run in the man page, README and TESTING.md. Tests: test_dtoctl.sh checks the new options, the --stats-run plan (--dry-run), a real stats run of a shell command and a failing one; tests/test_stats_report.py checks the report's arithmetic on made-up runs with known answers, sorts made-up perf lines, and finds memcpy, memset and memcmp by address in this machine's libc.
The first example reused 256 KiB buffers that stayed in the core's L2, the one case where the CPU beats DSA, and pre-faulted them with MAP_POPULATE, which on gnr-qual1's kernel makes about a quarter of DSA calls 5x slower. Replace it with a run that copies out of a 1 GiB buffer, pinned to the DSA's node: 1.20x measured, 1.21x predicted.
- A failed profiled run (perf not permitted, or a perf shim without a matching kernel perf) ended --stats-run with perf's status and no report, although every timed run had completed. Report them and say the profile is missing; the report's note no longer claims later runs did not happen when the failed run is a profiled one. - The report mixed a mean with medians: mem* time and bytes were averaged over the runs while the CPU and wall times were medians, so one preempted CPU-only run could push f past 100% or make "outside mem* calls" negative. Take the CPU time, mem* time and histogram from the run with the median CPU time; the wall time stays the median. The test gives its outlier run three times the mem* time to cover this. - restore_preload() put back the caller's whole LD_PRELOAD, so with a libdto exported in the caller's shell perf and the report ran with it, and the report appended a JSON summary to its own output. Restore the list without libdto entries. - libdto expands only %p in DTO_STATS_FILE, so refuse %p in the stats directory rather than any %, and check before creating the directory. - The shell test's "stats run succeeds" assertion needs python3 for the report; guard it like the output check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #38 (dtoctl) and #41 (stats). Only the last commit,
dtoctl: add --stats-run to measure the speedup from DSA, is new here. I'll rebase once those are merged.dtoctl --stats-runanswers two questions: how much faster is a program with DSA, and how much faster could it be? It runs the program CPU-only (DTO_USESTDC_CALLS=1) and then with DSA, collecting #41's JSON stats from each run.dto-stats-reportthen compares the runs using Amdahl's law. With--perf, it also profiles one run each way and shows where the CPU time went.This is gnr-qual1 with one DSA (wq0.0). The test program is single-threaded: it copies, sets and compares 256 KiB buffers that stay in cache, computes a software CRC32C, and does other work. The CPU-only run spent f = 31.5% of its CPU time in mem* calls, and those calls ran at s = 0.38x with DSA. Amdahl's law therefore predicts 0.66x, and the measured speedup was 0.63x. The remaining gap shows up on its own row: time outside mem* calls went from 663 to 748 ms. The profile shows the cost of DSA here: 34% of the DSA run's CPU time is DTO busy-polling.
What it does
dto-stats-<date>-<time>. It runs the program N times each way (--repeat), alternating the two modes so that drift affects both. For each run it records wall time, user+sys CPU time (wait4, so children are included) and the exit status inruns.jsonl, then runsdto-stats-report DIR.system(). A run that fails stops the rest, and dtoctl exits with its status.--dry-runprints the plan without running anything.--perf[=PERF]adds one more run each way underPERF record -e cpu-clock -F 999.cyclesgave about 10 samples a run.perf record -- dtoctl -l LIB -- prog), so DTO is preloaded into the program but not into perf.--jsonprints the numbers instead of the tables. It:stdc_onlyat exit), or every call belowmin_bytes;DTO_MIN_BYTESto try when there is a clear crossover;_exit()as dash does);.symtab, so perf shows__memmove_avx512_unaligned_ermsas a bare address. On the machine that made the profile, the report uses dlsym in its own process to find which memcpy/memset/memcmp implementations this CPU uses. It then places samples by address, but only after checking the profile's libc has the same build ID. Without this, the libc rows of the profile above would be unnamed.--stats-formatand--stats-intervalsetDTO_STATS_FORMATandDTO_STATS_INTERVAL_MS.Limits
sudorun needsudo dto-stats-report. The report says this instead of failing with a traceback.Testing
ctest -R "dtoctl|stats|functional"passes:tests/test_dtoctl.shchecks:--dry-runplan;runs.jsonl, the saved stats and the report;tests/test_stats_report.py(newstats-reportctest; needs neither DSA nor perf) checks:Running a stats run on RHEL 8 turned up a bug in #41: libdto wasn't linked with pthread. The fix is now on #41 (
stats: link libdto with pthread) and included here.