A hands-on guide for AI developers who want to run small agentic, speech and vision models on the device today. We install Cactus Compute's open-source engine, make a tool call, add confidence gating and a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, then package it all to run with the network switched off. Every example below was run on our own hardware, and we show what it printed, including where the models got things wrong.
pip install cactus-compute gives you three small models on one native engine: Needle 3 (35.3 MB) turns a request into a call to one of your Python functions or extracts fields from text; Whistle (16.9 MB) transcribes speech in seven European languages; Mirage (21.6 MB) returns objects, text and simple form answers from an image. They run on a CPU, offline after the first download. In our runs a tool call took about 70 to 110 ms with 136 MB peak memory. The models are narrow and sometimes wrong, so production use rests on three habits: strict schemas, a confidence threshold chosen from your own test set, and an escalation path to a larger model or a person.
This technote accompanies our blog post Sub-50 MB on-device AI: what Cactus's engine means for business, which covers the business case. One correction first: this engine is sometimes reported as an NVIDIA product. It is not. It is built by Cactus Compute, a Y Combinator company, and released under Apache 2.0 [1][9]. Facts and versions are as of 16 November 2026.
1. Who this is for, what you will build, what you need
Who it is for. Python developers and ML engineers who want AI features that work without a network round trip: voice commands in a field app, a kiosk that reads documents, a support tool that turns a message into a ticket, or a device that must keep working offline.
What you will build. A first tool call; a confidence-gated agent that acts, asks or escalates to an OpenAI-compatible endpoint; extraction that refuses invented values; speech to a tool call; image text reading; a latency benchmark and an evaluation harness; and an offline container.
Prerequisites. Python 3.9 or later [1]; macOS on Apple silicon, Linux (x86-64 or Arm64) or Windows, among the platforms Cactus lists [4]; about 200 MB of free disk for the package, engine and all three models; internet access for the first download only. No GPU is needed. We used an Apple M4 Max with macOS 26.7, Python 3.15 and cactus-compute 4.2.2 (engine 4.1.1), plus Docker for the Linux arm64 container.
2. How the pieces fit
The Python package is a thin layer: it turns your functions into JSON schemas, runs the agent loop, checks that arguments are grounded in the input and reports a confidence score. It calls one native engine library, which loads a separate weights file (.cact) per model [2]. On first use the engine and weights are downloaded from Hugging Face at pinned commits, checked against SHA-256 checksums and cached under ~/.cache/cactus [2][3].
Two things to know before you start. First, Needle 3 does not write prose. It returns a tool call, or an empty list when the request is off topic, which is a refusal [2]. If you need written answers, you need a bigger model (see the comparison in section 13). Second, Cactus describes the engine as "sub-1 MB" [2]; the uncompressed libraries we downloaded were 1.4 MB on macOS and 1.8 MB on Linux Arm64. Each model plus the engine is still well under 50 MB.
3. Step 1: install
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install "cactus-compute==4.2.2"
# Optional: switch off anonymous usage counts for this shell
export CACTUS_TELEMETRY=0
The pin matters: the package went from 4.0.0 to 4.2.3 in eleven days in November 2026 [8]. import cactus does not pull in JAX; JAX is only needed for the optional fine-tuning extra, cactus-compute[train] [2]. The package sends anonymous usage counts (function name, version, OS and a random install id) unless CACTUS_TELEMETRY=0 or DO_NOT_TRACK=1 is set [1][2].
4. Step 2: your first tool call
This is the README example [1], extended to show both ways of driving the agent.
import json
import cactus
@cactus.tool
def get_weather(city: str):
"Get the current weather for a city."
return {"city": city, "temp_c": 27, "sky": "clear"}
with cactus.Cactus(tools=[get_weather]) as agent:
# One turn: the model proposes a call, you decide whether to run it
r = agent.complete("What's the weather in Lagos?")
print(json.dumps({k: r[k] for k in ("type", "function_calls", "confidence", "peak_ram_mb")}))
# Full loop: Cactus runs your function and returns its result
agent.reset()
print(agent.run("What's the weather in Lagos?")["results"])
Run it with python hello.py. The first run downloads the engine and the 35 MB weights (about 10 seconds for us); later runs load from the cache in well under a second. Our output:
{"type": "call", "function_calls": [{"name": "get_weather", "arguments": {"city": "Lagos"}}], "confidence": 0.9555, "peak_ram_mb": 134.2}
[{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]
What happened:
@cactus.toolturns the function signature and docstring into a JSON schema. Type hints become argument types; a default makes an argument optional [2].complete()runs one turn and returns the proposed call without executing anything. Use it when you want to check or confirm before acting.run()executes the loop: the model picks calls, Cactus runs your functions, feeds results back and returns them underresults[2].reset()clears the conversation but keeps the tools. One toolset per agent: to change tools, create a newCactus[2].
5. Step 3: design tools a small model can use
A 35 MB model reads your whole toolset as part of its prompt and has no retrieval step, so tool design drives accuracy. Cactus recommends five tools or fewer, plain names without shared prefixes, closed choices as enums and bounded numbers [2]. The engine compiles your schemas into a grammar, so every argument it returns matches the schema; tighter schemas therefore leave less room to be wrong.
- Use
typing.Literal[...]for closed sets (rooms, priorities, categories). - Bound numbers with
Annotated[int, cactus.Field(ge=0, le=100)][2]. - Write the docstring as one plain sentence, with
Args:lines for anything ambiguous. - Pass facts such as the date and locale through
system=, for example"date: 2026-11-16 Mon 10:00; locale: en-GB", so "tomorrow at 7" can resolve [2].
6. Step 4: confidence gating and a fallback
Every response carries a confidence score between 0 and 1. The engine already moves calls below 0.1 into suppressed_calls; above that, the threshold is your product decision [2]. The script below acts at or above 0.7, asks for confirmation below it, and sends refusals to an OpenAI-compatible chat endpoint you control (a vLLM or llama.cpp server on your own GPU box, or a cloud API where the data allows). With no endpoint configured, it hands the request to a person.
import json
import os
import urllib.request
from typing import Literal
import cactus
CONFIDENCE_FLOOR = 0.7 # tune per product on your own test set
@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
"""Set a room's light brightness.
Args:
room: which room
brightness: brightness in percent, 0 to 100
"""
return {"room": room, "brightness": brightness}
@cactus.tool
def create_ticket(summary: str, priority: Literal["low", "normal", "urgent"] = "normal"):
"""Open a support ticket.
Args:
summary: one-line description of the problem
"""
return {"ticket": "T-1001", "summary": summary, "priority": priority}
TOOLS = {"set_lights": set_lights, "create_ticket": create_ticket}
def cloud_fallback(text: str) -> dict:
"""Send the request to an OpenAI-compatible endpoint you control, if configured."""
url = os.environ.get("FALLBACK_URL") # e.g. http://gpu-box:8000/v1/chat/completions
if not url:
return {"route": "human", "note": "no fallback configured; ask the user to confirm"}
body = json.dumps({
"model": os.environ.get("FALLBACK_MODEL", "default"),
"messages": [{"role": "user", "content": text}],
}).encode()
req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=20) as resp:
answer = json.load(resp)["choices"][0]["message"]["content"]
return {"route": "cloud", "answer": answer}
def handle(agent, text: str) -> dict:
agent.reset()
r = agent.complete(text)
calls, held = r["function_calls"], r.get("suppressed_calls") or []
conf = r["confidence"]
if calls and conf is not None and conf >= CONFIDENCE_FLOOR:
results = [TOOLS[c["name"]](**c["arguments"]) for c in calls]
return {"route": "local", "confidence": conf, "results": results}
if calls or held:
return {"route": "confirm", "confidence": conf, "proposed": calls or held}
return cloud_fallback(text)
if __name__ == "__main__":
with cactus.Cactus(tools=list(TOOLS.values()),
system="date: 2026-11-16 Mon 10:00; locale: en-GB") as agent:
for text in [
"dim the kitchen to 20 percent",
"the printer on floor 2 is on fire, raise it as urgent",
"write me a haiku about invoices",
]:
print(text, "->", json.dumps(handle(agent, text)))
Our output, with no fallback configured:
dim the kitchen to 20 percent -> {"route": "local", "confidence": 0.9622, "results": [{"room": "kitchen", "brightness": 20}]}
the printer on floor 2 is on fire, raise it as urgent -> {"route": "confirm", "confidence": 0.3689, "proposed": [{"name": "set_lights", "arguments": {"room": "lounge", "brightness": 100}}]}
write me a haiku about invoices -> {"route": "human", "note": "no fallback configured; ask the user to confirm"}
The second line is the reason for the gate. The model picked the wrong tool entirely, but with low confidence, so the user would have seen a confirmation instead of the lounge lights coming on. The third request was correctly refused. Set FALLBACK_URL (for example http://gpu-box:8000/v1/chat/completions) and FALLBACK_MODEL to route refusals to a larger model.
7. Step 5: structured extraction that refuses invented values
cactus.extract() declares your record as the only tool and returns the fields. In strict mode (the default) it raises ExtractionValidationError if a value is not grounded in the text [2].
import cactus
INVOICE = {
"name": "invoice",
"description": "An invoice record",
"parameters": {
"type": "object",
"properties": {
"vendor": {"type": "string"},
"total": {"type": "number"},
"currency": {"type": "string"},
},
"required": ["vendor", "total"],
},
}
for text in [
"Invoice from Acme Ltd. Amount due: 1200 GBP.",
"Invoice from Acme Ltd, total GBP 1,200.00, due 30 November.",
]:
try:
print("ok:", cactus.extract(text, INVOICE))
except cactus.ExtractionValidationError as err:
print("rejected:", err)
print(" unchecked value would have been:", cactus.extract(text, INVOICE, strict=False))
ok: {'vendor': 'Acme Ltd. Amount', 'total': 1200.0}
rejected: extraction returned values not grounded in the input: total
unchecked value would have been: {'vendor': 'Acme Ltd', 'total': 12200.0, 'currency': 'GBP'}
Two lessons from one small test. Strict mode caught a wrong total (12,200 instead of 1,200) that would otherwise have gone into your system. And the first result passed strict mode with a messy vendor name ("Acme Ltd. Amount"), because grounding checks that a value comes from the text, not that it is the right span. Keep your own validation for anything that matters.
8. Step 6: speech
Whistle needs 16 kHz mono WAV, or 16 kHz float samples. Make a test clip:
# macOS: synthesise a test clip and convert it to 16 kHz mono WAV
say -o lights.aiff "Turn the office lights down to forty percent."
afconvert -f WAVE -d LEI16@16000 -c 1 lights.aiff lights.wav
# Linux or Windows: convert any recording with ffmpeg
ffmpeg -i recording.m4a -ar 16000 -ac 1 lights.wav
import json
from typing import Literal
import cactus
@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
"""Set a room's light brightness.
Args:
room: which room
brightness: brightness in percent, 0 to 100
"""
return {"room": room, "brightness": brightness}
# 1. Plain transcription with word timestamps
t = cactus.transcribe("lights.wav", language="en", word_timestamps=True)
print(t["text"], "| ttft_ms:", t["ttft_ms"])
print([(w["word"] if "word" in w else w, w.get("start"), w.get("end")) for w in t["words"][:3]])
# 2. Speech straight to a tool call (no separate transcription step)
with cactus.Cactus(tools=[set_lights]) as agent:
r = agent.complete(audio="lights.wav", language="en")
print(r["audio_text"], "->", json.dumps(r["function_calls"]), r["confidence"])
Turn the office lights down to 40%. | ttft_ms: 298.1
[('Turn', 0.0, 0.24), ('the', 0.24, 0.4), ('office', 0.4, 0.64)]
Turn the office lights down to 40%. -> [{"name": "set_lights", "arguments": {"room": "office", "brightness": 40}}] 0.9515
Passing audio= to complete() goes from speech to a tool call in one step and returns the transcript as audio_text [2]. For a 5-second clip, transcription took 0.74 seconds once loaded. For live microphone input, cactus.stream() takes roughly one-second chunks and yields committed text as it goes; install cactus-compute[mic] for microphone capture and resampling [2].
Limits worth knowing: seven languages (English, German, French, Spanish, Italian, Dutch, Polish); silence returns empty text, but music and other noise can produce words, so put a voice activity detector or wake word in front of an always-on microphone [2]. Cactus reports Whistle beating Whisper base on several word error rate test sets and losing on others, at 16.9 MB against 145.3 MB [6]. We did not reproduce those benchmarks; our test clips were synthetic speech, which is easier than a real room.
9. Step 7: vision
import json
from PIL import Image, ImageDraw, ImageFont
from cactus import Mirage
# Make a test image so the example is self-contained
img = Image.new("RGB", (640, 400), "white")
draw = ImageDraw.Draw(img)
font = ImageFont.load_default(size=40)
draw.text((40, 40), "INVOICE 1042", fill="black", font=font)
draw.text((40, 120), "Total: GBP 1,200.00", fill="black", font=font)
draw.ellipse((420, 220, 560, 360), fill="red")
img.save("card.png")
vision = Mirage()
# Text lines with confidence (boxes are on a 0-1000 grid)
for line in vision.describe("card.png", sections=["text"])["text"]:
print(f'{line["score"]:.2f} {line["string"]} {line["bbox"]}')
# Fill a small form with closed choices
form = {
"document_type": {"enum": ["invoice", "receipt", "letter", "other"]},
"has_red_circle": {"enum": ["yes", "no"]},
}
for field, answer in vision.extract("card.png", form)["form"].items():
print(field, "->", answer["value"])
0.38 INVOICE [66, 113, 321, 210]
0.70 INVOICE 1042 [60, 108, 464, 216]
0.50 1042 [321, 109, 472, 216]
0.37 Total: GBP [55, 304, 369, 420]
0.67 Total: GBP 1,200.00 [52, 303, 620, 423]
0.51 1,200.00 [381, 310, 612, 418]
document_type -> invoice
has_red_circle -> None
Text reading worked, with overlapping candidates at different confidence levels; keep the highest-scoring line per region. Boxes come back on a 0 to 1,000 grid, not in pixels (our 640 by 400 image's red circle came back near x 650 to 880). The form answered one question and abstained on the other (None, with the probabilities in the full result). In an earlier run with a different font it answered the red-circle question wrongly. Mirage is useful for reading text and rough layout; for decisions, validate on your own images and gate on the returned probabilities.
10. Step 8: measure latency and memory
import os
import statistics
import time
import cactus
cactus.set_threads(int(os.environ.get("CACTUS_THREADS", "1")))
@cactus.tool
def set_lights(room: str, brightness: int):
"Set a room's light brightness in percent."
return {"room": room, "brightness": brightness}
QUERIES = ["dim the kitchen to 30 percent", "office lights to 80", "turn the lounge down to 10"]
t0 = time.perf_counter()
agent = cactus.Cactus(tools=[set_lights])
print(f"load: {time.perf_counter() - t0:.2f}s")
agent.complete(QUERIES[0]) # warm-up, not counted
wall, decode, ram = [], [], []
for i in range(30):
agent.reset()
t = time.perf_counter()
r = agent.complete(QUERIES[i % len(QUERIES)])
wall.append((time.perf_counter() - t) * 1000)
decode.append(r["decode_tps"])
ram.append(r["peak_ram_mb"])
agent.close()
wall.sort()
print(f"p50 {statistics.median(wall):.0f} ms p95 {wall[int(0.95 * len(wall)) - 1]:.0f} ms")
print(f"decode median {statistics.median(decode):.0f} tok/s peak RAM {max(ram):.0f} MB")
Run it with CACTUS_THREADS=1 python bench.py, then with 0 (all cores) and 4. Two of our runs:
CACTUS_THREADS=1: p50 105 ms p95 112 ms decode median 451 tok/s peak RAM 136 MB
CACTUS_THREADS=0: p50 81 ms p95 167 ms decode median 534 tok/s peak RAM 136 MB
Our Mac was shared with other heavy jobs, which proved useful. When other work saturated the cores, the default setting pushed the median to 1.4 seconds and four threads gave a 22-second p95, while one thread stayed at about 110 ms. A phone that is also rendering UI can hit the same contention. Start with one or two threads and measure on the slowest device you support. peak_ram_mb in each response is the engine's own figure; the whole Python process peaked at about 176 to 197 MB in our runs.
11. Step 9: package it to run offline
Nothing should download on a customer's device or in production. Fetch the engine and weights at build time and point the package at them [3].
A container that runs with networking off
FROM python:3.12-slim
ENV CACTUS_TELEMETRY=0 \
XDG_CACHE_HOME=/opt/cache \
PIP_NO_CACHE_DIR=1
RUN pip install "cactus-compute==4.2.2"
# Fetch the engine and the weights at build time; nothing downloads at run time.
RUN cactus fetch \
&& cactus download needle3 --out /opt/models \
&& chmod -R a+rX /opt/cache /opt/models
ENV HF_HUB_OFFLINE=1 \
CACTUS_NEEDLE_WEIGHTS=/opt/models/needle3.cact
RUN useradd --create-home app
USER app
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]
docker build -t cactus-edge-demo .
docker run --rm --network none cactus-edge-demo
We built this on Apple silicon (so a Linux arm64 image of about 134 MB) with gated.py as app.py, and it printed the same three routes as step 4 with networking disabled. One trap we hit: cactus download needle3 alone writes into the current directory, so without --out and CACTUS_NEEDLE_WEIGHTS the non-root user tried to download the weights again at run time and failed on a read-only cache.
An air-gapped Linux device
# On a connected machine with the same OS, CPU architecture and Python as the device
pip download "cactus-compute==4.2.2" -d wheels
cactus fetch --platform-tag manylinux2014_aarch64 --out ./engine # Linux Arm64 engine
cactus download needle3 --out ./models
# Copy wheels/, engine/ and models/ to the device, then on the device:
pip install --no-index --find-links wheels cactus-compute
export HF_HUB_OFFLINE=1 CACTUS_TELEMETRY=0
export CACTUS_LIB_PATH=$PWD/engine/libcactus.so
export CACTUS_NEEDLE_WEIGHTS=$PWD/models/needle3.cact
We checked the equivalent on macOS with an empty cache directory, HF_HUB_OFFLINE=1 and the two environment variables set: the tool call ran without touching the network. Use CACTUS_WHISTLE_WEIGHTS and CACTUS_MIRAGE_WEIGHTS for the other models [3].
Mobile and browser apps
For apps outside Python, cactus download <platform> --model needle3 fetches a native build: iOS, tvOS and watchOS get a static library and a C header (cactus.h); Android and Linux also get a command-line runner; the browser target is WebAssembly [5]. We did not build a mobile app for this note; follow Cactus's deployment guide and treat it as a C library you wrap in Swift or Kotlin.
12. Test and evaluate before you ship
The only accuracy figures that matter are the ones on your requests. Write a test file of real queries with the expected call, including ones that should be refused, then score it at several thresholds:
{"query": "dim the kitchen to 20 percent", "expect": {"name": "set_lights", "arguments": {"room": "kitchen", "brightness": 20}}}
{"query": "the wifi in the office keeps dropping", "expect": {"name": "create_ticket"}}
{"query": "what's the capital of France?", "expect": null}
import json
import sys
import cactus
from gated import TOOLS # reuse the tool definitions
def matches(call, expect):
if expect is None:
return call is None
if call is None or call["name"] != expect["name"]:
return False
wanted = expect.get("arguments", {})
return all(call["arguments"].get(k) == v for k, v in wanted.items())
cases = [json.loads(line) for line in open(sys.argv[1])]
rows = []
with cactus.Cactus(tools=list(TOOLS.values()), system="locale: en-GB") as agent:
for case in cases:
agent.reset()
r = agent.complete(case["query"])
call = r["function_calls"][0] if r["function_calls"] else None
rows.append((r["confidence"] or 0.0, call is not None, matches(call, case["expect"]), case))
for threshold in (0.0, 0.5, 0.7, 0.9):
acted = [row for row in rows if row[1] and row[0] >= threshold]
wrong = [row for row in acted if not row[2]]
missed = sum(1 for row in rows if row[3]["expect"] is not None and row not in acted)
print(f"threshold {threshold:.1f}: acted on {len(acted)}/{len(rows)}, "
f"wrong actions {len(wrong)}, not acted (sent to confirm/fallback) {missed}")
for conf, has_call, ok, case in rows:
print(f"{conf:.2f} {'ok ' if ok else 'BAD'} {case['query']}")
On our seven-case file (save it as cases.jsonl beside gated.py and run python evaluate.py cases.jsonl):
threshold 0.0: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.5: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.7: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
threshold 0.9: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
0.95 ok dim the kitchen to 20 percent
0.97 ok office lights to 75
0.96 ok make the lounge brighter, 90 percent
0.22 BAD the wifi in the office keeps dropping
0.62 ok log an urgent ticket: server room is overheating
0.96 ok what's the capital of France?
0.90 ok write me a poem
Seven cases prove nothing statistically; 100 to 200 real ones per tool surface is a sensible start. The shape is what matters: raising the threshold trades actions for confirmations. Pick the point where wrong actions are rare enough for the task, and rerun the file in CI whenever you change the package version, weights or tool definitions. Cactus ships six ready-made tool surfaces with acceptance suites in cactus.environments, and notes that the base model does not pass five of them at the default gate [2], a useful reminder that a small model's fit depends on the tools.
If the base model is not accurate enough on your tools, Cactus supports LoRA fine-tuning (a cheap way to adapt a model by training small extra weight matrices) locally with cactus-compute[train], or on its paid hosted platform, producing a new .cact you load with weights= [2]. Note that locally fine-tuned weights do not include the calibration head, so confidence comes back as None [2]; you would need another gate.
13. Running it in production
- Pin everything. Package version, and the weights file you tested with. Weekly releases are normal for this project.
- Threads. Set
cactus.set_threads()explicitly, before loading models; start at one or two on shared or mobile CPUs. - Concurrency. Text, speech and vision share one runtime and calls are serialised across threads [3]. For a server handling many requests, run several worker processes rather than threads.
- Security. The model chooses which of your functions to call. Only expose functions that are safe to run with any schema-valid arguments, and re-check arguments and permissions in the function itself.
- Monitoring. Log route (local, confirm, fallback), confidence, latency,
peak_ram_mband whether the user corrected the action. Avoid logging transcripts or extracted personal data unless you need them and have a basis to keep them. - Privacy. Disable telemetry where your policy requires it, and remember that the fallback path sends data off the device: apply the same data rules there.
14. Pitfalls and debugging
- Empty
function_callseverywhere. The request does not map to a tool, or the tool description is unclear. Check thereasoningfield and rewrite the docstring. confidenceisNone. You loaded locally fine-tuned weights without the calibration head [2].ExtractionValidationError. A value was not found in the text. Do not switch tostrict=Falseto make it go away; inspect it.- PermissionError on start in a container. The package is trying to download into a cache it cannot write. Bundle the files and set
CACTUS_NEEDLE_WEIGHTSandHF_HUB_OFFLINE=1. - Latency spikes. CPU contention with multi-threading. Try
set_threads(1). - Wrong sample rate. Whistle expects 16 kHz mono; convert, or install the
[mic]extra for resampling [1].
15. How it compares with the alternatives
| Option | What you run | Typical size | Best for | Trade-offs |
|---|---|---|---|---|
Cactus (cactus-compute) | Cactus's own Needle, Whistle and Mirage models on one engine | 17 to 35 MB per model | Narrow tool calls, extraction, short speech, OCR on any CPU, offline | No free text; small tool sets; young project; few independent benchmarks |
| llama.cpp (and Ollama on top) | Open LLMs and some vision models in GGUF format [11] | Hundreds of MB to tens of GB | General chat, drafting and tool use with 1B to 70B+ models on laptops and servers | Far more memory; tool-calling reliability varies by model |
| MLC LLM | Open LLMs compiled for phones, browsers and GPUs [14] | Hundreds of MB and up | Chat-class models inside mobile and web apps | Compilation step per model and target |
| ONNX Runtime | Your own model exported to ONNX [12] | Whatever you export | Shipping a custom-trained model across platforms and accelerators | You own training, export and pre/post-processing |
| ExecuTorch | Your PyTorch model, exported for edge devices; 1.0 since October 2025 [13] | Whatever you export | PyTorch teams targeting phones and embedded NPUs | More engineering; you bring the model |
| NVIDIA TensorRT Edge-LLM | LLMs, vision-language and speech models on Jetson, DRIVE and DGX Spark [10] | GB-class models | Robots, vehicles and edge boxes with NVIDIA GPUs; NVIDIA's own Nemotron models | NVIDIA hardware only; engines built per device |
NVIDIA's own on-device direction is larger models on NVIDIA edge hardware: its Nemotron 3 Nano Omni, for example, is a 30B-parameter mixture-of-experts model with about 3B active per token, aimed at Jetson and DGX Spark among other targets [15]. That is a different class from a 35 MB CPU model. A sensible architecture often uses both: a tiny model on the user's device for the frequent, narrow requests, and a larger model on a GPU server you control for the fallback.
16. FAQ
Is this an NVIDIA product? No. Cactus Compute builds and publishes it under Apache 2.0, including the prebuilt runtime binaries [1]. The nearest NVIDIA link is that Cactus also ships a Mac runner, cactus-mlx, for compressed checkpoints of NVIDIA's Nemotron models [1].
Can Needle answer questions or summarise text? No. It returns tool calls or extracted fields only. Use a larger model for prose.
Which languages does speech support? English, German, French, Spanish, Italian, Dutch and Polish; fine-tuning can add others [2].
Can I use it commercially? The code and runtime binaries are Apache 2.0 [1], and the Mirage model card lists Apache 2.0 [7]. Check the licence of each model you ship on its Hugging Face card.
If you are deciding which AI work should run on devices, on your own GPU servers or in the cloud, and need the private side built and run, see how we approach private AI.
References
- cactus-compute/cactus: README, GitHub, Cactus Compute.
- Cactus API reference (llms.txt), Cactus Compute.
- Needle Python API and CLI documentation, Cactus Compute.
- Needle supported devices, Cactus Compute.
- Cactus runtime binaries and platforms, Hugging Face.
- Whistle announcement, Cactus Compute, 2 October 2026.
- Mirage model card, Hugging Face.
- cactus-compute on PyPI (release history).
- Cactus Compute company page, Y Combinator.
- TensorRT Edge-LLM overview, NVIDIA.
- llama.cpp, GitHub.
- ONNX Runtime, Microsoft.
- Introducing ExecuTorch 1.0, PyTorch.
- MLC LLM, GitHub.
- NVIDIA Nemotron 3 Nano Omni, NVIDIA blog, 28 April 2026.
All code in this note was run by Workstation on 16 November 2026 with cactus-compute 4.2.2 on an Apple M4 Max and in a Linux arm64 container; outputs are copied from those runs. Performance figures are indicative, from a shared machine. Other figures are vendor claims. This note is independent and not endorsed by Cactus Compute or NVIDIA. NVIDIA, Jetson, DRIVE, DGX, Nemotron and TensorRT are trademarks of NVIDIA Corporation. Other names are the property of their respective owners.