Single-binary, scale-to-zero ML & deep learning inference daemon written in Rust.

Why SnoozeML? • Quickstart • Supported Formats • Python SDK • Configuration • Documentation

---

SnoozeML (snooze) is a bare-metal inference server engineered for high hardware utilization and zero idle memory waste. It combines adaptive dynamic micro-batching with in-process scale-to-zero memory reclamation, lazily loading model weights on demand and automatically evicting inactive models from GPU VRAM and host RAM.

Note

SnoozeML is purpose-built for predictive machine learning, deep learning encoders, computer vision, audio, and tabular models. Generative Large Language Models (LLMs) requiring autoregressive token decoding and KV-cache management are explicitly out of scope.

Why SnoozeML?

Most production inference stacks force a painful compromise: either wrestle with multi-gigabyte containers, Python dependency hell, and mandatory Kubernetes/Knative clusters just to scale idle models down, or pay for idle RAM/VRAM 24/7.

SnoozeML delivers enterprise-grade serving primitives in a single, rootless Rust binary:

Capability	Python Inference Frameworks	Heavyweight GPU Serving Suites	SnoozeML (snooze)
Deployment Footprint	Heavy pip/Conda envs + Docker	Multi-GB container images	Single static binary (zero sudo / Docker required)
Scale-to-Zero	Requires Kubernetes + Knative / KEDA	External orchestrator required	Native in-process idle timer & LRU eviction
Idle Baseline RAM	Hundreds of MBs (Python interpreter + workers)	High baseline footprint	Near-zero (reclaims 100% of model RAM/VRAM on idle)
Execution Engine	Python GIL workarounds / multi-process	C++ (complex build & plugin chains)	Native Rust (ort, candle, hessboost) + isolated Python IPC
Event Brokers	Requires external consumer glue code	Requires external microservices	Built-in Kafka, RabbitMQ, and Redis Streams consumers

Core Capabilities

• In-Process Scale-to-Zero & LRU Eviction: Unloads idle models after a configurable timeout, enforces max_loaded_models capacity budgets via LRU eviction, and recovers reactively from out-of-memory (OOM) spikes.
• Adaptive Dynamic Micro-Batching: Coalesces concurrent HTTP, gRPC, and broker stream requests along Axis(0) within a configurable latency budget (--max-delay-ms) and batch cap (--max-batch-size).
• Model Pinning & Replica Autoscaling: Pin mission-critical models in memory while dynamically scaling worker replicas under queue pressure (LeastQueue, RoundRobin, Random) with differential scale-down timeouts.
• Multi-Runtime Execution: Serve ONNX (ort), Hugging Face Safetensors (candle), XGBoost / GBDT trees (hessboost in pure Rust), and custom Python models (process-isolated workers that reclaim 100% of memory upon eviction) from one daemon.
• Unified Transports & Zero-Trust Security: Simultaneous HTTP REST (:8080), native gRPC (:50051), and Event Stream consumers with shared Rustls TLS, constant-time SHA-256 API keys, HMAC JWTs, and 3-tier RBAC (Admin, Inference, ReadOnly).
• Built-in Observability: Interactive terminal UI dashboard (snooze monitor), Prometheus metrics (/metrics), and health probes (/health/live, /health/ready).

---

Quickstart (3 Steps)

Get a Hugging Face embedding model (BAAI/bge-small-en-v1.5) running end-to-end in under two minutes.

1. Install SnoozeML

Linux & macOS (Zero sudo Required):

# Standard build (all backends, all streams, and CUDA/TensorRT/CoreML GPU support)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh

# Lightweight CPU-only build (smaller download, no GPU provider libraries)
curl -fsSL https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.sh | sh -s -- --cpu

Installs snooze and bundled ONNX Runtime into ~/.local/bin, initializes ~/.local/share/snoozeml/models, and registers a rootless background service (-- --cpu for CPU-only variant, -- --no-service to skip daemon, -- --uninstall to remove).

Windows (PowerShell):

# Standard build (includes CUDA/TensorRT GPU support)
irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iex

# Lightweight CPU-only build
$env:SNOOZE_VARIANT="cpu"; irm https://raw.githubusercontent.com/SnoozeML/SnoozeML/main/install.ps1 | iex

Note

Bare-Metal NVIDIA GPU Prerequisites: To keep binary downloads small (~50 MB instead of ~2 GB), prebuilt binaries dynamically link against your host's CUDA 13 Runtime (libcudart / libcublas 13) and cuDNN 9 (libcudnn 9) libraries (e.g. sudo apt-get install cuda-runtime-13-0 libcudnn9-cuda-13 on Ubuntu/Debian). With device = "auto" (the default), SnoozeML automatically logs a warning and falls back to CPU if CUDA libraries are absent; if device = "cuda" is explicitly configured, it returns an explicit error with setup instructions. For a zero-setup GPU environment with CUDA 13 + cuDNN 9 pre-bundled, use the ghcr.io/snoozeml/snoozeml:cuda container image.

From Source: (Rust 1.75+):

git clone https://github.com/SnoozeML/SnoozeML.git && cd SnoozeML
cargo build --release

Profile	Command	Description
Default	cargo build --release	ONNX, Candle Safetensors, XGBoost trees, Python workers, and streaming
NVIDIA CUDA	cargo build --release --features cuda	Enables CUDA execution provider
NVIDIA TensorRT	cargo build --release --features tensorrt	Enables TensorRT execution provider
Apple CoreML	cargo build --release --features coreml	Enables CoreML execution provider
All Brokers	cargo build --release --features streams-all	Enables Kafka, RabbitMQ, and Redis Streams drivers
Trees Only	cargo build --release --no-default-features --features tree-backend	Ultra-lightweight pure-Rust XGBoost build

2. Download a Model

Download BAAI/bge-small-en-v1.5 from Hugging Face Hub and auto-generate ./models/models.toml:

snooze download BAAI/bge-small-en-v1.5 --include "*.safetensors,config.json,tokenizer.json"

3. Serve & Run Inference

Start the server (if not using the background service) and send a request:

snooze serve --models-dir ./models

curl -s -X POST http://127.0.0.1:8080/v1/models/bge-small-en-v1.5/infer \
  -H "Content-Type: application/json" \
  -d '{
    "inputs": [{ "name": "input_ids", "shape": [1, 4], "data": [101.0, 7592.0, 2088.0, 102.0] }]
  }' | jq .

SnoozeML lazily loads bge-small-en-v1.5 on the first request, runs the forward pass, and returns the 384-dimensional embedding tensor. After 300 seconds of inactivity, weights are automatically deallocated back to idle.

---

Supported Model Formats

Drop any of the following into --models-dir for automatic discovery and registration:

Format	Extensions / Layout	Engine	Best For
ONNX	.onnx	ort (CPU, CUDA, TensorRT, CoreML)	PyTorch, Scikit-Learn, TensorFlow, and general deep learning
Safetensors	.safetensors + config.json	candle	Native Rust execution of BERT, RoBERTa, JinaBERT, NomicBERT, DistilBERT, CLIP, ViT, DINOv2, MLPs
XGBoost / GBDT	.xgb.json, .tree.json, .xgb	hessboost	Pure-Rust, zero-FFI tabular tree inference
Python Worker	Directory with snooze_model.py	Process-isolated binary IPC	Custom architectures (e.g., Chronos-Bolt) that cannot export to ONNX

Create models/<model-name>/snooze_model.py. Workers run in isolated child processes so scale-to-zero unloads reclaim 100% of host RAM and GPU VRAM:

from typing import Any, Dict
import numpy as np

class Model:
    def load(self, model_dir: str, device: str) -> None:
        self.device = device  # Load custom PyTorch / HF weights here

    def predict(self, inputs: Dict[str, np.ndarray]) -> Dict[str, Any]:
        return {"output": inputs["input"] * 2.0}

---

Python SDK & Model Export

Install the official snoozeml Python SDK directly from PyPI:

# Core lightweight HTTP REST client (zero compiled C-extensions)
pip install snoozeml

# Native gRPC inference + NumPy array support
pip install "snoozeml[grpc]"

# Complete suite: gRPC, NumPy, scikit-learn, ONNX, and XGBoost converters
pip install "snoozeml[full]"

Dependency Extras

SnoozeML uses modular package extras so you only install what your service needs:

Extra	Command	Included Packages	Use Case
(None / Base)	pip install snoozeml	requests	Ultra-lightweight HTTP client for model lifecycle management (load, unload, pin, unpin), status polling, health probes, and metrics. Zero compiled C-extensions.
[grpc]	pip install "snoozeml[grpc]"	grpcio, protobuf, numpy	High-throughput, low-latency binary gRPC inference with automatic NumPy array serialization.
[converters]	pip install "snoozeml[converters]"	numpy, scikit-learn, onnx, skl2onnx, xgboost[-cpu]	Offline model export tools (export_sklearn, export_xgboost) targeting ONNX and pure-Rust tree formats.
[full]	pip install "snoozeml[full]"	grpc, converters, worker	All-in-one installation for client, gRPC, custom workers, and model converters with zero CUDA bloat.
[onnx]	pip install "snoozeml[onnx]"	numpy, scikit-learn, onnx, skl2onnx	Scikit-learn estimator and pipeline export to ONNX runtime format.
[xgboost]	pip install "snoozeml[xgboost]"	numpy, xgboost-cpu (Linux) / xgboost (macOS/Win)	XGBoost model export to .xgb.json for pure-Rust hessboost tree execution. Uses xgboost-cpu on Linux to avoid pulling heavy NVIDIA CUDA packages.
[torch]	pip install "snoozeml[torch]"	torch	PyTorch nn.Module export to ONNX. (Note: If PyTorch is already installed in your training environment, snoozeml automatically duck-types PyTorch tensors without needing this extra).
[worker]	pip install "snoozeml[worker]"	numpy	Minimal dependency profile for process-isolated custom Python worker models (snooze_model.py).
[dev]	pip install "snoozeml[dev]"	pytest, mypy, ruff, sphinx	Development, linting, testing, and documentation building.

Note

For editable development from a cloned repository, use pip install -e "./python[full,dev]".

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from snoozeml import SnoozeClient
from snoozeml.converters import export_sklearn

# 1. Train and export directly to ./models
X, y = load_iris(return_X_y=True)
clf = RandomForestClassifier(n_estimators=50, random_state=42).fit(X, y)
export_sklearn(clf, initial_types=X[:1].astype(np.float32), output_path="models/iris_rf.onnx")

# 2. Pin in memory and query over gRPC
with SnoozeClient(endpoint="127.0.0.1:50051", use_grpc=True) as client:
    client.models.pin("iris_rf")
    response = client.infer("iris_rf", inputs={"float_input": X[:4].astype(np.float32)})
    print("Predictions:", response.get_numpy("label"))

Documentation available at https://snoozeml.github.io/SnoozeML/python/.

---

CLI, Configuration & Security

Manage lifecycle states (idle, loading, ready, busy) and live telemetry from the CLI or REST API:

snooze status --endpoint http://127.0.0.1:8080    # Inspect states & queue depths
snooze load --model-id bge-small-en-v1.5 --pin    # Warm up & exempt from scale-to-zero
snooze unpin --model-id bge-small-en-v1.5         # Re-enable idle eviction
snooze unload --model-id bge-small-en-v1.5        # Immediately reclaim RAM/VRAM
snooze monitor --endpoint http://127.0.0.1:8080   # Launch interactive Ratatui TUI

Endpoint	Method	Role	Description
/v1/models	GET	ReadOnly	List all models, residency states, and configuration
/v1/models/{id}/infer	POST	Inference	Execute dynamically micro-batched inference
/v1/models/{id}/{load,unload,pin,unpin}	POST	Admin	Manage model memory residency and pinning
/health/live, /health/ready	GET	Public	Liveness and readiness probes
/metrics	GET	Public / Admin	Prometheus metrics (snoozeml_requests_total, latency, batches)

Declarative Configuration (models.toml)

Generated via snooze init ./models or snooze download. Never store plaintext secrets in models.toml; map API keys and HMAC JWT secrets via environment variables (SNOOZE_API_KEY, SNOOZE_API_KEYS, SNOOZE_JWT_SECRET, or custom key_env mappings):

[defaults]
idle_timeout_secs = 300
max_batch_size = 16
max_batch_delay_ms = 10
device = "auto"
max_loaded_models = 8
autoscale_queue_threshold = 32
max_replicas = 4
routing_strategy = "least_queue" # "least_queue" | "round_robin" | "random"

[models."bge-small-en-v1.5"]
path = "bge-small-en-v1.5/model.safetensors"
pinned = true
max_batch_size = 64
max_batch_delay_ms = 5

[auth]
enabled = true
protect_metrics = true

[[auth.api_keys]]
name = "production-client"
key_env = "PROD_CLIENT_API_KEY"
role = "inference" # "readonly" | "inference" | "admin"

---

Event Streaming & Docker Deployment

Native Broker Consumers (--features streams-all)

Connect directly to Apache Kafka (rskafka), RabbitMQ (lapin), and Redis Streams (redis). Consumers feed straight into DynamicBatcher alongside HTTP/gRPC traffic with dynamic reply_to routing, commit-after-infer at-least-once delivery, and Dead-Letter Queue (dlq_topic) support.

[[streams]]
id = "kafka-embeddings"
driver = "kafka"
endpoint = "127.0.0.1:9092"
model_id = "bge-small-en-v1.5"
input_topic = "inference.requests"
output_topic = "inference.responses"
dlq_topic = "inference.dlq"
format = "json" # "json" | "protobuf"
concurrency = 32
group_id = "snooze-workers"

Official Container Images (GitHub Container Registry)

Official prebuilt, hardened multi-arch container images are published to the GitHub Container Registry (ghcr.io) on every release:

ghcr.io/snoozeml/snoozeml:<tag>

Three container options are available to match your deployment needs:

Option / Flavor	Primary Tags	Supported Platforms	Features & Backends
cuda (Default)	:latest, :cuda, :full, :<version>	linux/amd64	Full GPU-accelerated runtime with NVIDIA CUDA and TensorRT execution providers, Candle Safetensors, Tree models, bundled Python worker runtime (numpy, requests, snoozeml), and all streaming brokers (streams-all).
cpu	:cpu, :<version>-cpu	linux/amd64, linux/arm64	Standard CPU container (~83 MB compressed) with all model backends, bundled Python worker runtime, and all streaming brokers. Ideal for cloud CPU instances.
slim	:slim, :lite, :<version>-slim	linux/amd64, linux/arm64	Ultra-lean (~52 MB compressed) pure Rust runtime. Includes ONNX Runtime (CPU), Candle, and Tree backends with zero Python dependencies for minimal CVE attack surface.

Pull & Run with Docker

# Pull the default GPU container (CUDA + TensorRT + Python workers)
docker pull ghcr.io/snoozeml/snoozeml:latest

# Or pull the standard CPU image (~83 MB compressed)
docker pull ghcr.io/snoozeml/snoozeml:cpu

# Or pull the minimal slim image (~52 MB compressed, zero Python)
docker pull ghcr.io/snoozeml/snoozeml:slim

# Launch container with local models directory mounted:
docker run -d \
  --name snoozeml \
  # --gpus all \                 # Pass GPU to container (NVIDIA Container Toolkit required)
  -p 8080:8080 -p 50051:50051 \
  -v ./models:/models:ro \
  ghcr.io/snoozeml/snoozeml:latest

Tip

For GPU acceleration with the default cuda image, ensure the NVIDIA Container Toolkit is installed and pass --gpus all to docker run.

Build from Source Locally

You can also compile custom images directly using Docker BuildKit and cargo-chef:

docker build --target cuda -t snooze-ml:cuda .  # Default full GPU build
docker build --target cpu -t snooze-ml:cpu .    # Standard CPU + Python build
docker build --target slim -t snooze-ml:slim .  # Lean pure-Rust build (~52 MB compressed)

# Or launch via Docker Compose / Buildx Bake
docker compose up snooze-cuda

---

Documentation, Examples & Quality Assurance

Documentation is available at https://snoozeml.github.io/SnoozeML/.

• Interactive Examples: Step-by-step Jupyter notebooks and custom worker templates (Scikit-Learn ONNX, pure-Rust XGBoost trees, Amazon Chronos-Bolt, and real-time Kafka event streaming) are located in the examples/ directory.
• Documentation Portal: Run npm --prefix docs run dev to launch the local Astro Starlight developer portal, or build the complete documentation bundle with npm --prefix docs run build and sphinx-build -b html python/docs docs/dist/python.
• Deep Dives: Architectural and operational specifications are available in docs/architecture.md, docs/api.md, docs/configuration.md, and the Production Deployment Guide.
• Strict QA: All production Rust code enforces #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic)]. Verify with cargo test --all-targets, cargo clippy --all-targets -- -D warnings, and pytest python/tests -v.

---

Roadmap

SnoozeML is actively evolving. The following initiatives represent planned architectural enhancements and future milestones:

• gRPC Streaming: Server-streaming (InferStream) and bidirectional channels for real-time telemetry, audio, and video ingestion.
• WASM & Native Plugins: Zero-copy in-process pre/post-processing via cdylib and WebAssembly (Wasmtime/WASI) without Python IPC overhead.
• Edge & Embedded Targets: Cross-compilation for RISC-V/ARM64, strict host memory ceilings, and WebGPU (wgpu) / Vulkan / Apple Metal acceleration.
• KServe v2 Data Plane Parity: Drop-in REST/gRPC protocol compatibility for existing cloud-native ML meshes.
• Model Versioning: Multi-version model coexistence, semantic version pinning, and route aliasing.
• Hot-Swap & Canary Deployments: Zero-downtime model reloading and dynamic traffic splitting for gradual rollouts.
• Rust Support for System 1 Models: Expanded native Rust architectures and kernels for fast decision models.
• In-Engine DAG Pipelines: Multi-model chaining with zero serialization hops between stages.

---

Platform Compatibility

Note

Linux Compatibility & Legacy glibc Notice: Prebuilt Linux release binaries target glibc >= 2.35 and statically link C++ runtime dependencies (e.g., Ubuntu 22.04 LTS, Ubuntu 24.04 LTS, Debian 12+, Fedora 36+, WSL2). Legacy enterprise distributions such as RHEL 8 (glibc 2.28) are not supported by prebuilt binaries directly—on RHEL 8 or older distributions, please either deploy via the official container images (ghcr.io/snoozeml/snoozeml:cpu or :cuda) or build from source with system toolchains.

---

Acknowledgements

SnoozeML builds upon outstanding projects in the Rust and Machine Learning ecosystems:

• ort & ONNX Runtime for cross-platform ONNX execution and hardware acceleration.
• candle, safetensors, tokenizers, and hf-hub by Hugging Face.
• hessboost & XGBoost for pure-Rust gradient boosted decision tree inference.
• Tokio, Axum, Tonic, Rayon, ndarray, and Ratatui.

---

License

Licensed under the Apache-2.0 License.