diff --git a/rayforce/README.md b/rayforce/README.md index 3dcdde7729..a9655b7e93 100644 --- a/rayforce/README.md +++ b/rayforce/README.md @@ -1,165 +1,38 @@ # Rayforce [Rayforce](https://github.com/RayforceDB/rayforce) is a zero-dependency -columnar analytics and graph engine written in pure C (MIT). Columnar -scans and graph traversals share one operation DAG, which is optimized -and then executed as morsel-driven bytecode over 1024-row batches. -Queries are written in **Rayfall**, Rayforce's Lisp-like query language, -so `queries.sql` holds 43 Rayfall expressions (one per line) rather than -SQL. +columnar analytics and graph engine written in C (MIT). Queries are +written in Rayfall, its Lisp-like query language, so `queries.sql` holds +43 Rayfall expressions rather than SQL. ## Install -`./install` builds from source with `make release`. Rayforce publishes an -amd64 `.deb` and prebuilt tarballs, but ClickBench also runs on aarch64 -machines and the engine is dependency-free C that builds with a plain -`make` on both, so building is the only setup that covers every machine -in the matrix. `RAYFORCE_VERSION=vX.Y.Z ./install` pins a release; -unset, it takes the latest GitHub release. +`./install` builds the latest release from source with +`make release CC=gcc`, which works on both amd64 and aarch64. +`RAYFORCE_VERSION=vX.Y.Z ./install` pins a release. ## Data layout -`./load` streams `hits.parquet` into a splayed on-disk table under -`./hits` with `.parquet.splayed` — one file per column plus the `.sym` -dictionary, written row group by row group without materializing the -table in memory. The column names come from the file; `create.rfl` -gives the native type of each column in the file's order. - -Types follow `../clickhouse/create.sql`: `BIGINT` → `I64`, `INTEGER` → -`I32`, `SMALLINT` → `I16`, `TIMESTAMP` → `TIMESTAMP`, `Date` → `DATE`. -The Parquet file keeps `EventDate` as integer days since 1970-01-01 and -`EventTime`, `ClientEventTime` and `LocalEventTime` as integer Unix -seconds; their entries in the type vector, `UNIX_DATE` and -`UNIX_SECONDS`, make the reader convert them to Rayforce's native date -and timestamp types, so `EventDate >= '2013-07-01'`, -`extract(minute FROM EventTime)` and `DATE_TRUNC('minute', EventTime)` -all work on native values. - -### Why every text column is SYM - -Rayforce has two text column types: `SYM` (dictionary-encoded, one -integer index per row into a global intern table) and `STR` -(variable-length, 12 bytes inline or a per-vector byte pool). `STR` is -the type its docs recommend for high-cardinality text such as URLs, but -it does not reach ClickBench scale in the current release: - -* **The pool offset is a `uint32_t`**, so a column's pool is capped at - 4 GiB (`src/io/csv.c` bails out above it). At 100M rows the `URL`, - `Title`, `Referer` and `OriginalURL` columns hold roughly 9, 11, 8 and - 5 GB of bytes, so none of them fit in one `STR` column. -* **Opening a `STR` column validates every element** (bounds plus a - 4-byte prefix compare against the pool, `col_validate_str_region`), - and the cost is superlinear: on a subset of this dataset a splayed - table with five `STR` columns opened in 0.4 s at 2M rows but 199 s at - 9.6M rows. The same table with dictionary-encoded text opened in - 17 s at 9.6M rows. - -Loading all TEXT / VARCHAR / CHAR columns as `SYM` sidesteps both: it -has no 4 GiB limit, and the columns become narrow integer vectors (the -9.6M-row subset is 4.4 GB with every text column dictionary-encoded, -versus 7.5 GB with those five columns as `STR`). -Dictionary encoding is what makes the dataset loadable at all here; it -also makes `GROUP BY URL` an integer group-by, while the string -operations in Q28 and Q29 pay an extra indirection per row to resolve -symbol ids back to bytes. A third reason not to go back to `STR` for -now: `take:` — i.e. every `LIMIT` — currently returns empty strings for -pool-backed values -([RayforceDB/rayforce#404](https://github.com/RayforceDB/rayforce/issues/404)). +`./load` streams `hits.parquet` into a compressed splayed table under +`./hits` with `.parquet.splayed`. Types follow +`../clickhouse/create.sql`; text columns are `SYM` (dictionary-encoded). +On machines with more RAM than the Parquet file the dictionaries share +one `.sym` file, otherwise each text column gets its own compressed +symbol file. ## Server mode -Rayforce is embeddable and normally invoked as a CLI, but this entry runs -it as an IPC server (`./start` → `rayforce -p 5000 server.rfl`) because -opening the table is eager: it validates every column file and loads the -symbol dictionary before the first query. Paying that once per server -start instead of once per query process keeps it out of the reported -numbers and out of the run's wall clock. The listening socket only -accepts connections after `server.rfl` finishes, so `./check` — a -one-expression IPC round trip — is a genuine readiness probe. - -`./start` runs once before `./load`, when `./hits` does not exist yet, and -again before every query, when it must be there. `./start` tells the two -cases apart with `RAYFORCE_REQUIRE_TABLE`, and in the second one a failed -open is fatal: `server.rfl` exits instead of serving. Otherwise the server -would sit there with `hits` bound to `0`, answer the readiness probe -happily, and return `1` from query 1 — `(count hits)` — in microseconds, -which the driver would record as a valid timing rather than an error. - -`./start` and `./stop` match the server with -`pgrep -x -f "rayforce -p server.rfl"` rather than by process name: -`./query` and `./check` spawn clients that are *also* called `rayforce`, so -a `-x rayforce` name match cannot tell a live client from a live server. It -made `./start` a no-op whenever a client happened to be in flight — most of -the time during the concurrent-QPS test — so the watchdog could silently -fail to restart a crashed server, and it made `./stop` kill the workers' -in-flight queries as well. Anchoring with `-x -f` matches the whole -command line, so it picks out the server without the usual `-f` hazard of -also matching a shell that merely mentions it. - -`./query` sends the query text; the server evaluates -`(timeit (set rf-result ))`, which returns the elapsed -milliseconds from a nanosecond clock, and a second untimed round trip -pulls the result back so it can be printed. Timing therefore covers -server-side query execution only, the same convention as the other -embedded engines here (e.g. DuckDB's `.timer`). - -### Cold runs - -`BENCH_RESTARTABLE=yes` is required, not optional: with the server left -running, `drop_caches` cannot evict pages that a live process still has -mapped, and a "cold" query measured 0.029 s — exactly its warm time — -versus 0.44 s after a real restart. Restarting between queries makes the -cold number honest, at the price of re-reading the table on every start -(hence `BENCH_CHECK_TIMEOUT=1800`). Note the flip side: because the open -is eager, on a machine whose RAM comfortably exceeds the dataset the -first query still runs against a fully resident table. +`./start` runs `rayforce -p 5000 server.rfl`, which opens the table once +per start and writes `server.ready`; `./check` looks for that file and a +live server pid. `./query` sends the query over IPC and the server times +it with `timeit`, so the timing covers query execution only. The driver +restarts the server before every query, so cold runs are cold. ## Query adaptations -The queries are direct Rayfall translations of the ClickBench SQL. All 43 -were checked against `clickhouse-local` on a 2M-row subset of the same -CSV; the only differences are which of several equally-ranked rows a -`LIMIT` over ties returns, plus the projection differences noted below. -Points worth knowing: - -* **`COUNT(*)`** is `(count hits)`; `COUNT(*)` over a filter is - `(count (select {...}))`. -* **Ungrouped `COUNT(DISTINCT c)`** (Q5, Q6) is a whole-column reducer, - `(count (distinct (at hits 'c)))`, and not a `select` projection: the - projection form returns the row count instead of the distinct count - ([RayforceDB/rayforce#405](https://github.com/RayforceDB/rayforce/issues/405)). - Per-group `COUNT(DISTINCT c)` (Q9-Q12, Q14, Q23) is written inline as - `(count (distinct UserID))` and lowers to Rayforce's grouped - count-distinct kernels. -* **`HAVING`** (Q28, Q29) has no clause form: the group-by runs as an - inner `select` and the outer one filters on the aggregate. -* **`LIMIT n OFFSET m`** (Q39-Q43) is `take: [m n]`. -* **`extract(minute FROM EventTime)`** (Q19) is `(minute EventTime)`; - **`DATE_TRUNC('minute', EventTime)`** (Q43) is - `(xbar EventTime 60000000000)`, i.e. truncation to whole minutes of - the nanosecond timestamp. -* **`CASE WHEN ... END`** (Q40) is the row-wise `(if cond then else)`, - which the query compiler lowers to the DAG's ternary select. -* **`ORDER BY` a column that is not selected** (Q25, Q27): sorting - happens after projection, so those queries project `EventTime` - alongside `SearchPhrase`. Same rows, same order, one extra column in - the printed output. -* **`GROUP BY 1, URL`** (Q35): a constant is not accepted as a group - key, and grouping by `(1, URL)` is the same partition as grouping by - `URL`, so the constant is projected by an outer `select` instead. -* **Derived group keys** (Q36) go in the `by:` dict as - `ip1: (- ClientIP 1)`, which the optimizer turns into a synthetic - column rather than a materialized one. Rayforce emits the aggregate - before the derived keys, so the result has the same rows as the SQL - with the columns in a different order. -* **`REGEXP_REPLACE`** (Q29): Rayforce has no regex engine, so the - host extraction `^https?://(?:www\.)?([^/]+)/.*$` is spelled out with - the string builtins — `str-find` for `://`, `www.` and the first `/`, - `substr` to cut, and `if` to fall back to the whole `Referer` when the pattern - does not match (no `http`/`https` scheme at offset 0, or no `/` after - the host). On this dataset it reproduces `REGEXP_REPLACE` exactly, - including the fallback rows: the group keys, counts, `MIN(Referer)` - and average lengths all match ClickHouse. -* **`MIN(URL)` / `MIN(Title)`** (Q22, Q23) work directly on `SYM` - columns and return the lexicographic minimum, not the minimum - dictionary index. +The queries are direct Rayfall translations of the SQL, checked against +`clickhouse-local` on a 2M-row subset. Differences: which rows a `LIMIT` +over ties returns; Q25 and Q27 also print `EventTime` (sorting happens +after projection); Q36 puts the aggregate before the derived keys; Q29 +spells out `REGEXP_REPLACE` with string builtins and matches ClickHouse +on this dataset. diff --git a/rayforce/benchmark.sh b/rayforce/benchmark.sh index 47f4c87d8b..cd6a7862a0 100755 --- a/rayforce/benchmark.sh +++ b/rayforce/benchmark.sh @@ -1,11 +1,8 @@ #!/bin/bash export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single" -# Rayforce runs as an IPC server here (./start), so the stop/drop_caches/start -# cold cycle is meaningful and the concurrent-QPS test hits a shared process. +# IPC server: restarts make cold runs cold, and the concurrent test hits one process. export BENCH_RESTARTABLE=yes export BENCH_DURABLE=yes -# Opening the splayed table validates every column file and loads the symbol -# dictionary; at 100M rows that takes minutes, and it happens on every restart, -# so the readiness probe needs a much longer window than the 300s default. +# Every start opens the table, which takes up to about a minute. export BENCH_CHECK_TIMEOUT="${BENCH_CHECK_TIMEOUT:-1800}" exec ../lib/benchmark-common.sh diff --git a/rayforce/check b/rayforce/check index 67948aa8fc..e956ebd1d3 100755 --- a/rayforce/check +++ b/rayforce/check @@ -1,4 +1,9 @@ #!/bin/bash +# Ready = server.rfl has opened the table (server.ready) and the server is alive. +# +# No IPC probe: the server takes new connections only between queries, so in +# the concurrent test a probe waited behind the workers and the watchdog +# restarted a server that was busy, not dead. set -e -export RAYFORCE_PORT="${RAYFORCE_PORT:-5000}" -rayforce check.rfl >/dev/null + +[ -f server.ready ] && [ -f server.pid ] && kill -0 "$(cat server.pid)" 2>/dev/null diff --git a/rayforce/check.rfl b/rayforce/check.rfl deleted file mode 100644 index 4af950ee31..0000000000 --- a/rayforce/check.rfl +++ /dev/null @@ -1,11 +0,0 @@ -; Readiness probe: connect over IPC and evaluate a trivial expression. -; The server binds its socket before running server.rfl but only accepts -; connections once that script has finished opening the table, so a successful -; round trip means "table open, ready to query". -; The server answers one request at a time, so during the concurrent phase -; the handshake waits behind whatever query is running; the timeout must -; cover a long query, or a busy server is mistaken for a dead one. -(set h (.ipc.open (format "127.0.0.1:%" (.os.getenv "RAYFORCE_PORT")) 300000)) -(set ok (.ipc.send h "(+ 1 1)")) -(.ipc.close h) -(if (== ok 2) (exit 0) (exit 1)) diff --git a/rayforce/create.rfl b/rayforce/create.rfl index f61d471666..1657ec879d 100644 --- a/rayforce/create.rfl +++ b/rayforce/create.rfl @@ -1,16 +1,7 @@ -; ClickBench `hits` schema for Rayforce. -; -; `load` passes `hits-types` to `.parquet.splayed`, one entry per column -; of hits.parquet in file order; `hits-names` lists that order. -; -; Types follow ../clickhouse/create.sql: BIGINT -> I64, INTEGER -> I32, -; SMALLINT -> I16, TIMESTAMP -> TIMESTAMP, Date -> DATE. The Parquet -; file keeps EventDate as integer days since 1970-01-01 and EventTime, -; ClientEventTime and LocalEventTime as integer Unix seconds: UNIX_DATE -; and UNIX_SECONDS make the reader convert them to native DATE and -; TIMESTAMP. Every TEXT / VARCHAR / CHAR column is loaded as SYM -; (dictionary-encoded) rather than STR -- see README.md "Why every text -; column is SYM". +; ClickBench `hits` schema: `hits-names` is the column order of hits.parquet, +; `hits-types` the type of each column for `.parquet.splayed`. Types follow +; ../clickhouse/create.sql; UNIX_DATE and UNIX_SECONDS convert the integer +; dates and times of the Parquet file; text columns are SYM. (set hits-names [ WatchID JavaEnable Title GoodEvent EventTime EventDate CounterID ClientIP diff --git a/rayforce/data-size b/rayforce/data-size index a7bc3a6540..35894e89d7 100755 --- a/rayforce/data-size +++ b/rayforce/data-size @@ -1,5 +1,5 @@ #!/bin/bash set -e -# Splayed table: one file per column plus the `.sym` dictionary, all under ./hits. +# Column files and symbol files, all under ./hits. du -sb hits | cut -f1 diff --git a/rayforce/install b/rayforce/install index dcbb151f81..14766756ea 100755 --- a/rayforce/install +++ b/rayforce/install @@ -1,21 +1,12 @@ #!/bin/bash -# Builds Rayforce from source. +# Builds Rayforce from source (amd64 and aarch64). The Makefile defaults to +# clang; build-essential brings gcc. RAYFORCE_VERSION pins a tag, otherwise the +# latest release (or the fallback when the GitHub API is rate limited). # -# Rayforce publishes an amd64 .deb and prebuilt tarballs, but ClickBench also -# runs on aarch64 machines (c8g.*), and the engine is zero-dependency C that -# builds with a plain `make release` on both. Building from source is therefore -# the only setup that works everywhere, and it is what upstream documents first. -# -# RAYFORCE_VERSION pins the release tag; unset means "latest GitHub release" -# (falling back to a known-good tag when the unauthenticated API call is rate -# limited). -# -# Last benchmarked release: v2.11.5. Bump this line to request a new run: -# the benchmark automation reruns a system when a file in its directory -# changes. +# Last benchmarked release: v3.0.1. Bump this line to request a new run. set -e -RAYFORCE_FALLBACK_VERSION=v2.11.5 +RAYFORCE_FALLBACK_VERSION=v3.0.1 if ! command -v rayforce >/dev/null 2>&1; then sudo apt-get update @@ -33,7 +24,7 @@ if ! command -v rayforce >/dev/null 2>&1; then rm -rf "$HOME/rayforce-src" git clone --depth 1 --branch "$version" \ https://github.com/RayforceDB/rayforce "$HOME/rayforce-src" - make -C "$HOME/rayforce-src" release -j"$(nproc)" + make -C "$HOME/rayforce-src" release CC=gcc -j"$(nproc)" sudo install -m 755 "$HOME/rayforce-src/rayforce" /usr/local/bin/rayforce fi diff --git a/rayforce/load b/rayforce/load index 8d1f980c46..fa55cbd184 100755 --- a/rayforce/load +++ b/rayforce/load @@ -1,15 +1,21 @@ #!/bin/bash -# Loads hits.parquet into the splayed table ./hits. -# -# The load runs in its own rayforce process, not through the server: the server -# started by the driver before ./load has no table open yet and picks it up at -# the next restart, which happens before the first query. +# Loads hits.parquet into the splayed table ./hits, in its own process; the +# server picks the table up at its next restart. With more RAM than the +# Parquet file the text columns share one .sym file, otherwise each gets its +# own compressed symbol file. set -e rm -rf hits +mem=$(awk '/^MemTotal:/ { print $2 * 1024 }' /proc/meminfo) +if [ "$mem" -gt "$(stat -L -c %s hits.parquet)" ]; then + export HITS_SYMFILE=shared +else + export HITS_SYMFILE=column +fi +echo "MemTotal $mem bytes: $HITS_SYMFILE symbol file" + rayforce load.rfl -# The Parquet file is no longer needed once the columns are on disk. rm -f hits.parquet sync diff --git a/rayforce/load.rfl b/rayforce/load.rfl index bba9d764da..6f7020033d 100644 --- a/rayforce/load.rfl +++ b/rayforce/load.rfl @@ -1,9 +1,9 @@ -; Stream hits.parquet into a splayed on-disk table under ./hits. -; -; `.parquet.splayed` reads the file row group by row group and writes one column -; file per field plus the `.sym` dictionary, without materialising the whole -; table in memory. The column names come from the file; `hits-types` gives the -; native type of each column in the file's order. +; Stream hits.parquet row group by row group into a compressed splayed table +; under ./hits. `types` gives the column types in file order; HITS_SYMFILE +; (set by ./load) picks one shared .sym file or one symbol file per column. (load "create.rfl") -(.parquet.splayed hits-types "hits.parquet" "hits") +(set opts (if (== (.os.getenv "HITS_SYMFILE") "column") + (dict [types compress symfile] (list hits-types true 'column)) + (dict [types compress] (list hits-types true)))) +(.parquet.splayed opts "hits.parquet" "hits") (exit 0) diff --git a/rayforce/query b/rayforce/query index 34217395ee..fba25267cd 100755 --- a/rayforce/query +++ b/rayforce/query @@ -1,7 +1,6 @@ #!/bin/bash -# Reads a Rayfall query from stdin and runs it on the Rayforce server. -# Stdout: query result. Stderr: query runtime in fractional seconds on the last -# line. Exit non-zero on error. +# Runs a Rayfall query from stdin on the server. Stdout: the result; stderr: +# the runtime in seconds on the last line. Non-zero exit on error. set -e export RAYFORCE_PORT="${RAYFORCE_PORT:-5000}" @@ -14,14 +13,11 @@ cat > "$RAYFORCE_QUERY" rayforce query.rfl -# Guard against reporting a query that produced no timing as 0 seconds. ms=$(cat "$RAYFORCE_MS") if [ -z "$ms" ]; then echo "rayforce: query returned no timing" >&2 exit 1 fi -# `timeit` reports milliseconds and Rayforce may render them in exponent form -# (5.6e-06); awk normalises whatever it printed into plain fractional seconds, -# which is the only shape the driver's timing parser accepts. +# Milliseconds, possibly in exponent form, to plain fractional seconds. awk -v ms="$ms" 'BEGIN { printf "%.6f\n", ms / 1000 }' >&2 diff --git a/rayforce/query.rfl b/rayforce/query.rfl index 200a9a2cd9..35706508e7 100644 --- a/rayforce/query.rfl +++ b/rayforce/query.rfl @@ -1,13 +1,8 @@ -; Per-query client. Sends the Rayfall expression in $RAYFORCE_QUERY to the -; server, which times it with `timeit` (nanosecond clock, result in ms) and -; keeps the result in `rf-result`. A second, untimed round trip pulls the result -; back so ./query can print it. The elapsed milliseconds are written to -; $RAYFORCE_MS; ./query converts them to the fractional seconds the benchmark -; driver expects. -; The connect timeout also covers the wait behind other clients' queries -; (the server is single-threaded on IPC), so it spans the whole 600 s -; concurrent window rather than a fixed 60 s. -(set h (.ipc.open (format "127.0.0.1:%" (.os.getenv "RAYFORCE_PORT")) 600000)) +; Sends $RAYFORCE_QUERY to the server, which times it with `timeit` (ms) and +; keeps the result; a second, untimed round trip fetches the result. The +; milliseconds go to $RAYFORCE_MS. The connect timeout covers the wait behind +; other clients' queries in the concurrent test. +(set h (.ipc.open (format "127.0.0.1:%" (.os.getenv "RAYFORCE_PORT")) 3600000)) (set ms (.ipc.send h (format "(timeit (set rf-result %))" (read (.os.getenv "RAYFORCE_QUERY"))))) (show (.ipc.send h "rf-result")) diff --git a/rayforce/results/20261025/c6a.2xlarge.json b/rayforce/results/20261025/c6a.2xlarge.json new file mode 100644 index 0000000000..5bd61f551b --- /dev/null +++ b/rayforce/results/20261025/c6a.2xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c6a.2xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 733, + "data_size": 20944030989, + "concurrent_qps": 1.105, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.01, 0.001, 0.001], + [0.009, 0, 0], + [0.009, 0, 0], + [0.791, 0.554, 0.553], + [0.215, 0.096, 0.095], + [0.009, 0, 0], + [0.045, 0.011, 0.011], + [0.853, 0.49, 0.489], + [1.133, 0.74, 0.687], + [0.406, 0.144, 0.145], + [0.638, 0.22, 0.219], + [0.24, 0.107, 0.109], + [0.9, 0.402, 0.399], + [0.604, 0.174, 0.175], + [1.705, 1.139, 1.163], + [2.517, 1.608, 1.603], + [0.475, 0.038, 0.038], + [5.05, 2.209, 2.233], + [0.113, 0.006, 0.005], + [0.812, 0.354, 0.355], + [1.125, 0.155, 0.153], + [3.324, 0.543, 0.546], + [0.698, 0.029, 0.029], + [0.051, 0.002, 0.002], + [0.252, 0.198, 0.196], + [0.059, 0.002, 0.002], + [0.77, 0.434, 0.433], + [2.283, 1.864, 1.86], + [0.022, 0, 0.001], + [1.195, 0.256, 0.251], + [3.653, 0.278, 0.277], + [5.474, 2.848, 2.885], + [0.858, 0.555, 0.556], + [0.826, 0.55, 0.552], + [1.56, 1.081, 1.085], + [0.056, 0.013, 0.015], + [0.056, 0.013, 0.013], + [0.059, 0.006, 0.006], + [0.135, 0.076, 0.055], + [0.091, 0.009, 0.008], + [0.075, 0.008, 0.008], + [0.059, 0.012, 0.011] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c6a.4xlarge.json b/rayforce/results/20261025/c6a.4xlarge.json new file mode 100644 index 0000000000..47df809598 --- /dev/null +++ b/rayforce/results/20261025/c6a.4xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c6a.4xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 699, + "data_size": 20944030989, + "concurrent_qps": 3.455, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.012, 0.001, 0.001], + [0.014, 0, 0], + [0.014, 0, 0], + [0.434, 0.209, 0.209], + [0.201, 0.09, 0.091], + [0.014, 0, 0], + [0.061, 0.008, 0.007], + [1.4, 0.984, 0.974], + [1.765, 0.309, 0.309], + [0.331, 0.086, 0.086], + [0.366, 0.097, 0.097], + [0.231, 0.072, 0.072], + [0.771, 0.249, 0.247], + [0.689, 0.093, 0.092], + [1.312, 0.395, 0.395], + [2.287, 0.929, 0.916], + [0.448, 0.021, 0.021], + [4.314, 1.314, 1.303], + [0.115, 0.004, 0.004], + [0.753, 0.179, 0.178], + [1.13, 0.085, 0.085], + [3.293, 0.236, 0.235], + [1.036, 0.014, 0.014], + [0.065, 0.003, 0.003], + [0.164, 0.094, 0.094], + [0.077, 0.003, 0.003], + [0.758, 0.225, 0.224], + [1.613, 1.066, 1.065], + [0.033, 0, 0], + [1.421, 0.131, 0.131], + [3.917, 0.149, 0.147], + [4.908, 1.677, 1.711], + [0.637, 0.252, 0.263], + [0.624, 0.247, 0.256], + [1.199, 0.373, 0.372], + [0.08, 0.011, 0.011], + [0.08, 0.011, 0.011], + [0.091, 0.006, 0.006], + [0.144, 0.067, 0.046], + [0.135, 0.007, 0.007], + [0.111, 0.006, 0.007], + [0.083, 0.009, 0.008] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c6a.large.json b/rayforce/results/20261025/c6a.large.json new file mode 100644 index 0000000000..c9da89d0f1 --- /dev/null +++ b/rayforce/results/20261025/c6a.large.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c6a.large", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 1060, + "data_size": 16026815123, + "concurrent_qps": 0.377, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.022, 0.003, 0.003], + [0.01, 0, 0], + [0.01, 0, 0], + [1.758, 1.262, 1.263], + [0.416, 0.088, 0.088], + [0.01, 0, 0], + [0.074, 0.03, 0.03], + [3.455, 2.203, 2.233], + [3.628, 1.924, 1.928], + [1.229, 0.457, 0.454], + [1.483, 0.557, 0.553], + [0.733, 0.364, 0.367], + [3.096, 2.014, 2.045], + [1.347, 0.725, 0.722], + [3.033, 2.782, 2.374], + [4.177, 3.373, 3.307], + [1.372, 0.137, 0.134], + [8.96, 6.062, 6.205], + [0.361, 0.017, 0.015], + [4.617, 3.587, 3.562], + [2.592, 1.119, 0.877], + [9.143, 8.11, 8.468], + [0.932, 0.129, 0.122], + [0.116, 0.004, 0.004], + [1.69, 1.57, 1.57], + [0.12, 0.004, 0.004], + [2.491, 1.658, 1.662], + [13.281, 6.76, 6.778], + [0.024, 0, 0], + [2.404, 1.356, 1.962], + [5.709, 3.505, 3.049], + [14.588, 8.863, 8.807], + [1.763, 0.346, 0.346], + [1.713, 0.334, 0.332], + [4.018, 3.584, 3.597], + [0.07, 0.027, 0.027], + [0.06, 0.017, 0.017], + [0.058, 0.007, 0.007], + [0.365, 0.211, 0.184], + [0.103, 0.011, 0.011], + [0.079, 0.011, 0.01], + [0.083, 0.018, 0.017] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c6a.metal.json b/rayforce/results/20261025/c6a.metal.json new file mode 100644 index 0000000000..f9f01aaf39 --- /dev/null +++ b/rayforce/results/20261025/c6a.metal.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c6a.metal", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 220, + "data_size": 20973639597, + "concurrent_qps": 11.947, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.008, 0.001, 0.001], + [0.009, 0, 0], + [0.009, 0, 0], + [0.311, 0.141, 0.142], + [0.164, 0.108, 0.406], + [0.009, 0, 0], + [0.096, 0.008, 0.012], + [0.717, 0.263, 0.278], + [0.748, 0.104, 0.109], + [0.242, 0.057, 0.142], + [0.272, 0.068, 0.079], + [0.167, 0.039, 0.062], + [0.615, 0.185, 0.19], + [0.458, 0.048, 0.058], + [0.761, 0.167, 0.18], + [1.056, 0.172, 0.175], + [0.34, 0.006, 0.006], + [2.971, 0.295, 0.304], + [0.112, 0.004, 0.007], + [0.631, 0.086, 0.089], + [1.064, 0.056, 0.063], + [3.187, 0.15, 0.174], + [0.672, 0.016, 0.015], + [0.078, 0.001, 0.002], + [0.118, 0.035, 0.03], + [0.091, 0.004, 0.004], + [0.688, 0.133, 0.177], + [1.568, 0.996, 1.003], + [0.02, 0, 0], + [0.834, 0.065, 0.062], + [3.513, 0.057, 0.063], + [3.183, 0.31, 0.286], + [0.446, 0.181, 0.212], + [0.421, 0.148, 0.184], + [0.628, 0.137, 0.138], + [0.104, 0.012, 0.016], + [0.108, 0.01, 0.016], + [0.119, 0.015, 0.011], + [0.192, 0.069, 0.062], + [0.137, 0.011, 0.017], + [0.12, 0.015, 0.013], + [0.111, 0.01, 0.01] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c6a.xlarge.json b/rayforce/results/20261025/c6a.xlarge.json new file mode 100644 index 0000000000..9bcf56c476 --- /dev/null +++ b/rayforce/results/20261025/c6a.xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c6a.xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 894, + "data_size": 16026815123, + "concurrent_qps": 0.59, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.018, 0.002, 0.002], + [0.011, 0, 0], + [0.011, 0, 0], + [1.203, 0.732, 0.751], + [0.306, 0.087, 0.087], + [0.01, 0, 0], + [0.066, 0.018, 0.018], + [1.961, 1.095, 1.08], + [2.199, 1.401, 1.355], + [0.767, 0.226, 0.226], + [1.108, 0.658, 0.641], + [0.417, 0.176, 0.175], + [1.647, 0.92, 0.918], + [0.898, 0.312, 0.311], + [2.661, 1.886, 1.952], + [3.722, 2.508, 2.463], + [0.749, 0.067, 0.067], + [5.214, 3.457, 3.346], + [0.218, 0.009, 0.008], + [2.212, 1.684, 1.683], + [1.132, 0.414, 0.415], + [5.325, 1.022, 0.987], + [0.908, 0.064, 0.064], + [0.102, 0.002, 0.002], + [0.935, 0.84, 0.838], + [0.106, 0.003, 0.003], + [1.321, 0.796, 0.797], + [5.078, 4.382, 4.39], + [0.026, 0, 0], + [1.795, 0.468, 0.467], + [3.921, 0.502, 0.502], + [8.09, 5.122, 5.19], + [1.223, 0.673, 0.661], + [1.203, 0.667, 0.661], + [2.477, 2.091, 2.053], + [0.068, 0.017, 0.017], + [0.06, 0.014, 0.013], + [0.062, 0.006, 0.006], + [0.209, 0.14, 0.122], + [0.108, 0.009, 0.008], + [0.085, 0.008, 0.008], + [0.076, 0.012, 0.011] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c7a.metal-48xl.json b/rayforce/results/20261025/c7a.metal-48xl.json new file mode 100644 index 0000000000..652261797f --- /dev/null +++ b/rayforce/results/20261025/c7a.metal-48xl.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c7a.metal-48xl", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 220, + "data_size": 20939937381, + "concurrent_qps": 16.502, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.007, 0.001, 0.001], + [0.011, 0, 0], + [0.011, 0, 0], + [0.205, 0.036, 0.034], + [0.178, 0.097, 0.11], + [0.011, 0, 0], + [0.065, 0.009, 0.011], + [0.524, 0.088, 0.089], + [0.785, 0.06, 0.055], + [0.229, 0.032, 0.033], + [0.262, 0.035, 0.036], + [0.132, 0.035, 0.035], + [0.488, 0.068, 0.118], + [0.525, 0.031, 0.034], + [0.839, 0.076, 0.083], + [1.322, 0.092, 0.093], + [0.362, 0.003, 0.003], + [2.819, 0.115, 0.115], + [0.101, 0.007, 0.009], + [0.664, 0.032, 0.034], + [1.081, 0.034, 0.053], + [3.183, 0.129, 0.13], + [0.891, 0.007, 0.007], + [0.069, 0.001, 0.002], + [0.15, 0.015, 0.015], + [0.078, 0.004, 0.004], + [0.744, 0.093, 0.105], + [1.256, 0.717, 0.746], + [0.023, 0, 0], + [1.075, 0.034, 0.042], + [3.505, 0.036, 0.037], + [3.379, 0.138, 0.124], + [0.449, 0.084, 0.084], + [0.403, 0.084, 0.082], + [0.718, 0.07, 0.092], + [0.083, 0.013, 0.021], + [0.082, 0.013, 0.022], + [0.093, 0.012, 0.012], + [0.166, 0.072, 0.052], + [0.129, 0.014, 0.014], + [0.108, 0.012, 0.018], + [0.094, 0.008, 0.012] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c8g.4xlarge.json b/rayforce/results/20261025/c8g.4xlarge.json new file mode 100644 index 0000000000..e989943096 --- /dev/null +++ b/rayforce/results/20261025/c8g.4xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c8g.4xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 1155, + "data_size": 20944030989, + "concurrent_qps": 5.367, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.011, 0.001, 0.001], + [0.012, 0, 0], + [0.012, 0, 0], + [0.287, 0.067, 0.068], + [0.173, 0.08, 0.08], + [0.012, 0, 0], + [0.044, 0.006, 0.006], + [1.068, 0.563, 0.568], + [1.778, 0.21, 0.21], + [0.265, 0.045, 0.045], + [0.297, 0.049, 0.049], + [0.157, 0.042, 0.042], + [0.631, 0.1, 0.099], + [0.707, 0.055, 0.054], + [1.204, 0.177, 0.177], + [2.245, 0.651, 0.651], + [0.538, 0.01, 0.01], + [4.447, 0.886, 0.879], + [0.108, 0.003, 0.003], + [0.867, 0.107, 0.108], + [1.188, 0.058, 0.059], + [3.354, 0.126, 0.126], + [1.096, 0.01, 0.008], + [0.053, 0.003, 0.002], + [0.136, 0.059, 0.059], + [0.069, 0.004, 0.002], + [0.817, 0.131, 0.13], + [1.549, 0.636, 0.646], + [0.026, 0, 0], + [1.456, 0.072, 0.072], + [4.213, 0.077, 0.077], + [4.66, 0.95, 0.948], + [0.582, 0.166, 0.164], + [0.577, 0.162, 0.161], + [1.009, 0.157, 0.162], + [0.065, 0.009, 0.009], + [0.064, 0.009, 0.009], + [0.08, 0.005, 0.005], + [0.103, 0.041, 0.031], + [0.125, 0.006, 0.006], + [0.1, 0.005, 0.005], + [0.067, 0.006, 0.006] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/c8g.metal-48xl.json b/rayforce/results/20261025/c8g.metal-48xl.json new file mode 100644 index 0000000000..78281a7c86 --- /dev/null +++ b/rayforce/results/20261025/c8g.metal-48xl.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "c8g.metal-48xl", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 219, + "data_size": 20946620461, + "concurrent_qps": 21.223, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.005, 0.001, 0.001], + [0.006, 0, 0], + [0.006, 0, 0], + [0.192, 0.032, 0.031], + [0.137, 0.089, 0.088], + [0.006, 0, 0], + [0.033, 0.005, 0.005], + [0.587, 0.063, 0.063], + [0.79, 0.087, 0.087], + [0.135, 0.021, 0.02], + [0.2, 0.022, 0.022], + [0.087, 0.021, 0.022], + [0.497, 0.039, 0.038], + [0.356, 0.024, 0.024], + [0.56, 0.043, 0.043], + [0.92, 0.054, 0.054], + [0.419, 0.002, 0.002], + [3.09, 0.073, 0.073], + [0.056, 0.003, 0.003], + [0.717, 0.023, 0.023], + [1.125, 0.027, 0.027], + [3.229, 0.064, 0.063], + [0.589, 0.007, 0.007], + [0.04, 0.001, 0.001], + [0.058, 0.012, 0.012], + [0.071, 0.003, 0.002], + [0.768, 0.101, 0.1], + [0.984, 0.454, 0.43], + [0.013, 0, 0], + [0.742, 0.027, 0.028], + [3.653, 0.03, 0.029], + [3.15, 0.085, 0.085], + [0.429, 0.075, 0.075], + [0.428, 0.074, 0.097], + [0.495, 0.042, 0.042], + [0.046, 0.006, 0.006], + [0.045, 0.007, 0.007], + [0.056, 0.006, 0.006], + [0.088, 0.036, 0.026], + [0.071, 0.007, 0.007], + [0.058, 0.006, 0.006], + [0.05, 0.005, 0.004] +] + } + \ No newline at end of file diff --git a/rayforce/results/20261025/t3a.small.json b/rayforce/results/20261025/t3a.small.json new file mode 100644 index 0000000000..88aa76c080 --- /dev/null +++ b/rayforce/results/20261025/t3a.small.json @@ -0,0 +1,60 @@ +{ + "system": "Rayforce", + "date": "2026-10-25", + "machine": "t3a.small", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C","column-oriented","embedded"], + "load_time": 1700, + "data_size": 16026815123, + "concurrent_qps": 0.252, + "concurrent_error_ratio": 0, + "result": [ + [0, 0, 0], + [0.033, 0.006, 0.006], + [0.011, 0, 0], + [0.012, 0, 0], + [2.842, 2.039, 2.061], + [1.494, 1.125, 1.139], + [0.011, 0, 0], + [0.097, 0.053, 0.054], + [4.389, 2.508, 2.526], + [4.838, 3.134, 3.139], + [1.535, 0.644, 0.644], + [2.142, 0.862, 0.861], + [1.056, 0.722, 0.721], + [4.774, 3.537, 3.468], + [1.836, 1.116, 1.108], + [4.813, 4.057, 4.199], + [6.594, 5.697, 5.698], + [1.509, 0.18, 0.182], + [13.313, 10.222, 10.36], + [0.449, 0.028, 0.027], + [12.021, 12.591, 12.397], + [9.965, 10.113, 9.878], + [17.953, 18.698, 18.93], + [1.708, 0.213, 0.212], + [0.169, 0.005, 0.004], + [3.15, 3.024, 2.953], + [0.154, 0.004, 0.004], + [3.806, 2.56, 2.564], + [24.766, 24.596, 24.028], + [0.024, 0, 0], + [3.278, 1.735, 1.612], + [6.54, 2.068, 2.349], + [30.457, 28.032, 27.776], + [2.4, 0.68, 0.682], + [2.425, 0.684, 0.677], + [7.313, 6.758, 6.824], + [0.099, 0.045, 0.045], + [0.076, 0.031, 0.031], + [0.071, 0.013, 0.012], + [0.578, 0.414, 0.37], + [0.134, 0.025, 0.022], + [0.102, 0.018, 0.018], + [0.122, 0.037, 0.036] +] + } + \ No newline at end of file diff --git a/rayforce/server.rfl b/rayforce/server.rfl index 71a9357785..a284fe42c9 100644 --- a/rayforce/server.rfl +++ b/rayforce/server.rfl @@ -1,28 +1,15 @@ -; Init script for the Rayforce IPC server (see ./start). -; -; Opening the splayed table is eager: Rayforce validates every column file and -; loads the symbol dictionary up front, so this costs seconds-to-minutes on the -; 100M-row dataset. Doing it here, once per server start, keeps that cost out of -; the per-query numbers instead of paying it in every one of the 129 query -; invocations a benchmark run makes. -; -; ./start runs before ./load in the benchmark driver, so the table directory may -; not exist yet on the very first start; `try` keeps the server up in that case. -; The driver stops and starts the server again before each query, and that later -; start picks the table up. -; -; The listening socket only starts accepting after this script returns, which is -; exactly the readiness signal ./check needs. +; Server init script (see ./start): opens the table once per start, so the +; open stays out of the query timings. Before ./load there is no table yet and +; the server stays up without it. (set hits (try (.db.splayed.get "hits") (fn [e] (println "hits not loaded yet") 0))) (set hits-ready (== (type hits) 'TABLE)) (if hits-ready (println (format "loaded % rows" (count hits))) 0) -; Once ./hits exists on disk a failed open is fatal, and ./start says so with -; RAYFORCE_REQUIRE_TABLE=1. Staying up with `hits` bound to 0 would look healthy -; to any liveness probe, while query 1 — `(count hits)` — returned 1 in -; microseconds and got recorded as a valid timing (every other query would error -; out). Exiting instead keeps ./check failing, which the driver reports as a -; start failure rather than as benchmark results. +; Once ./hits exists, a failed open is fatal: serving with `hits` bound to 0 +; would record query 1 as a valid timing. (set fatal (and (not hits-ready) (== (.os.getenv "RAYFORCE_REQUIRE_TABLE") "1"))) (if fatal (do (println "./hits exists but could not be opened; refusing to serve") (exit 1)) 0) + +; Read by ./check. +(write "server.ready" "ready") diff --git a/rayforce/start b/rayforce/start index ac57aa999c..4f70f98e9a 100755 --- a/rayforce/start +++ b/rayforce/start @@ -1,43 +1,28 @@ #!/bin/bash -# Starts the Rayforce IPC server (see server.rfl for why the table is opened -# once here rather than in every query process). -# -# stdin/stdout are not a terminal, so rayforce skips the REPL and runs only the -# IPC poll loop. The listening socket accepts connections only after server.rfl -# returns, which makes ./check a real readiness probe. +# Starts the Rayforce IPC server; ./check reports it ready once server.rfl has +# opened the table. set -e port="${RAYFORCE_PORT:-5000}" -# Match the server's whole command line, not the process name. ./query and -# ./check each spawn a client that is also called `rayforce`, and during the -# concurrent-QPS test ten workers keep one of them alive essentially all the -# time, so a `pgrep -x rayforce` guard reported "already running" whenever a -# client happened to be in flight: the driver's watchdog could then silently -# fail to restart a crashed server and the rest of the throughput window would -# run against a dead process. `-x -f` anchors the pattern to the *entire* -# command line, so it picks out the server without also matching a shell that -# merely mentions it — the hazard that ruled out a plain `-f`. +# Match the whole command line: clients are also called `rayforce`. if pgrep -x -f "rayforce -p $port server.rfl" >/dev/null 2>&1; then exit 0 fi -# A server that has just been stopped keeps its listening socket open while -# it unmaps the table on exit, so a start issued right after ./stop could -# fail with "Address already in use". Wait for the port to be released. +# Wait for the port of a server that is still exiting. for i in $(seq 1 300); do ss -ltn 2>/dev/null | grep -q ":$port " || break sleep 1 done -# Tell server.rfl whether a table is supposed to be there: the driver starts -# the server once before ./load, when ./hits legitimately does not exist yet, -# and again before every query, when it must open successfully. +# The table must open on every start after ./load. if [ -d hits ]; then export RAYFORCE_REQUIRE_TABLE=1 else export RAYFORCE_REQUIRE_TABLE=0 fi +rm -f server.ready nohup rayforce -p "$port" server.rfl >> server.log 2>&1 & echo $! > server.pid diff --git a/rayforce/stop b/rayforce/stop index f81d23aa03..741029a170 100755 --- a/rayforce/stop +++ b/rayforce/stop @@ -1,26 +1,16 @@ #!/bin/bash -# Stops the Rayforce IPC server. The engine keeps its columns mmap'd, so the -# process has to be gone before the driver's drop_caches can evict them. -# -# Kill the recorded pid first, then sweep in case server.pid is missing or -# stale. The sweep matches the server's whole command line rather than the bare -# process name: ./query and ./check spawn clients that are also named -# `rayforce`, and `pkill -x rayforce` killed those too — during the -# concurrent-QPS test that meant the watchdog shot down the workers' in-flight -# queries on every poll. `-x -f` anchors the pattern to the entire command -# line, so unlike a plain `-f` it cannot match a shell that merely mentions it. +# Stops the server and waits until the process and its port are gone, so that +# drop_caches can evict the mapped columns. Matches the whole command line: +# clients are also called `rayforce`. port="${RAYFORCE_PORT:-5000}" +rm -f server.ready if [ -f server.pid ]; then kill "$(cat server.pid)" 2>/dev/null || true rm -f server.pid fi pkill -x -f "rayforce -p $port server.rfl" 2>/dev/null || true -# The signal only starts the shutdown: the process keeps the listening socket -# while it unmaps the table, and its command line is already cleared, so a -# ./start that follows immediately could bind neither the port nor see the -# process. Wait until both the process and the port are gone. for i in $(seq 1 300); do if ! pgrep -x -f "rayforce -p $port server.rfl" >/dev/null 2>&1 \ && ! ss -ltn 2>/dev/null | grep -q ":$port "; then