商店現已上線

我們打造用於網站、AI 智慧體和 LLM 訓練的 Workstation AI 機器——線上配置併購買。

獲取同行價格
AI 產品
AI 實驗室OpenAI 智慧體Claude 代理Grok BotWorkstation AI 原生 CRM營銷所有產品
AI 解決方案
應用型 AIAI 智慧體生成式 AIAI 自動化AI 技能提升中小企業 AI 套餐企業 AI 實驗室AI 工作站NVIDIA 硬體與軟體NVIDIA 加速計算私有 AIGPU 私有云GPU 叢集邊緣 AI行業 AI
AI 服務
平臺現代化數字工程資料基礎與 AI自主運營AI 諮詢DevOps 自動化網路安全軟體開發智慧體構建MLOps 搭建GPU 租賃GPU 私有云
關於我們
公司我們的故事合作伙伴客戶案例
AI 資料中心
AI 資料中心GPU 私有云GPU 叢集GPU 租賃
AI 洞察Jev 系列基準測試實驗室模型選擇
技術說明
文件
Linux 作業系統UbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian、Rocky 與 AlmaLinux
AI 雲平臺AWS 與 Amazon BedrockGoogle Cloud 與 Vertex AIMicrosoft Azure 與 AI FoundryGPU 雲與算力市場
私有 AI 資料中心面向律師事務所面向會計師事務所本地部署平臺全面安全的雲
工具Ring PromoterJobshout
新聞
部落格
聯絡我們登入

面向現代企業的 AI 工作站、AI 多智慧體軟體、GPU 基礎設施和智慧代理解決方案。

聯絡我們

AI 解決方案

應用型 AIAI 智慧體生成式 AIAI 自動化AI 技能提升中小企業 AI 套餐企業 AI 實驗室行業 AI

產品

所有產品Workstation AI native CRM 與 ERP營銷OpenAI 智慧體Ring PromoterJobshout

公司

關於我們公司AI 資料中心為什麼選擇Workstation合作伙伴客戶案例定價聯絡TenthMatrix

資源

AI 洞察技術說明文件Linux 作業系統AI 雲平臺私有 AI 資料中心部落格搜尋網站地圖
英國總部
77-79 Marlowes, Hemel Hempstead HP1 1LF路線指引 — 從 M25 外環倫敦 20 號出口駛出公司編號: 11641870週一至週五:上午 9:00 - 下午 6:00 GMT
+44 7515 356 146
比利時辦公室
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683週一至週五:上午 9:00 - 下午 6:00 CET
+32 492 45 67 46
印度辦公室
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
台灣辦公室
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓週一至週五:上午 9:00 - 下午 6:00 臺灣時間(GMT+8)
+886 926 892 701+32 492 45 67 46

© 2026 Workstation AI。保留所有權利。

隱私資料刪除Cookie服務條款網站地圖

NVIDIA、NVIDIA 標誌、RTX、RTX PRO、Blackwell、CUDA、NVLink、MGX、Omniverse、Spectrum-X、Quantum、BlueField、DGX 與 HGX 為 NVIDIA Corporation 在美國及其他國家的商標及/或註冊商標。AMD、AMD Arrow 標誌、EPYC、Instinct、Ryzen 與 Threadripper 為 Advanced Micro Devices, Inc. 的商標。Dell 與 PowerEdge 為 Dell Inc. 或其子公司的商標。Intel 與 Xeon 為 Intel Corporation 或其子公司的商標。AMAX、AceleMax 與 LiquidMax 為 AMAX Engineering Corporation 的商標。所有其他產品名稱、標誌與品牌均為其各自所有者之財產。我們使用它們僅為識別我們所描述、供應或支援的產品;使用這些名稱並不表示其所有者為我們背書。

首頁 / 技術說明 / Technology
AIAI AgentsPythonLLM

Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Install cactus-compute, make tool calls, gate on confidence with a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, and ship an offline container, with every example run and a comparison with llama.cpp, ONNX Runtime, ExecuTorch, MLC LLM and TensorRT Edge-LLM

2026年11月16日Technology22 min read

A hands-on guide for AI developers who want to run small agentic, speech and vision models on the device today. We install Cactus Compute's open-source engine, make a tool call, add confidence gating and a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, then package it all to run with the network switched off. Every example below was run on our own hardware, and we show what it printed, including where the models got things wrong.

A terminal running pip install cactus-compute and a JSON tool call, connected to a laptop, a phone and a single-board computer running the same model offline.

Summary. pip install cactus-compute gives you three small models on one native engine: Needle 3 (35.3 MB) turns a request into a call to one of your Python functions or extracts fields from text; Whistle (16.9 MB) transcribes speech in seven European languages; Mirage (21.6 MB) returns objects, text and simple form answers from an image. They run on a CPU, offline after the first download. In our runs a tool call took about 70 to 110 ms with 136 MB peak memory. The models are narrow and sometimes wrong, so production use rests on three habits: strict schemas, a confidence threshold chosen from your own test set, and an escalation path to a larger model or a person.

This technote accompanies our blog post Sub-50 MB on-device AI: what Cactus's engine means for business, which covers the business case. One correction first: this engine is sometimes reported as an NVIDIA product. It is not. It is built by Cactus Compute, a Y Combinator company, and released under Apache 2.0 [1][9]. Facts and versions are as of 16 November 2026.

1. Who this is for, what you will build, what you need

Who it is for. Python developers and ML engineers who want AI features that work without a network round trip: voice commands in a field app, a kiosk that reads documents, a support tool that turns a message into a ticket, or a device that must keep working offline.

What you will build. A first tool call; a confidence-gated agent that acts, asks or escalates to an OpenAI-compatible endpoint; extraction that refuses invented values; speech to a tool call; image text reading; a latency benchmark and an evaluation harness; and an offline container.

Prerequisites. Python 3.9 or later [1]; macOS on Apple silicon, Linux (x86-64 or Arm64) or Windows, among the platforms Cactus lists [4]; about 200 MB of free disk for the package, engine and all three models; internet access for the first download only. No GPU is needed. We used an Apple M4 Max with macOS 26.7, Python 3.15 and cactus-compute 4.2.2 (engine 4.1.1), plus Docker for the Linux arm64 container.

2. How the pieces fit

The Python package is a thin layer: it turns your functions into JSON schemas, runs the agent loop, checks that arguments are grounded in the input and reports a confidence score. It calls one native engine library, which loads a separate weights file (.cact) per model [2]. On first use the engine and weights are downloaded from Hugging Face at pinned commits, checked against SHA-256 checksums and cached under ~/.cache/cactus [2][3].

Architecture: your Python app, the cactus Python layer, the native libcactus engine and three weights files, with a first-use download from Hugging Face into a local cache.
One engine, three weights files, a thin Python layer. Sizes are the files we measured.

Two things to know before you start. First, Needle 3 does not write prose. It returns a tool call, or an empty list when the request is off topic, which is a refusal [2]. If you need written answers, you need a bigger model (see the comparison in section 13). Second, Cactus describes the engine as "sub-1 MB" [2]; the uncompressed libraries we downloaded were 1.4 MB on macOS and 1.8 MB on Linux Arm64. Each model plus the engine is still well under 50 MB.

3. Step 1: install

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install "cactus-compute==4.2.2"

# Optional: switch off anonymous usage counts for this shell
export CACTUS_TELEMETRY=0

The pin matters: the package went from 4.0.0 to 4.2.3 in eleven days in November 2026 [8]. import cactus does not pull in JAX; JAX is only needed for the optional fine-tuning extra, cactus-compute[train] [2]. The package sends anonymous usage counts (function name, version, OS and a random install id) unless CACTUS_TELEMETRY=0 or DO_NOT_TRACK=1 is set [1][2].

4. Step 2: your first tool call

This is the README example [1], extended to show both ways of driving the agent.

import json

import cactus


@cactus.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}


with cactus.Cactus(tools=[get_weather]) as agent:
    # One turn: the model proposes a call, you decide whether to run it
    r = agent.complete("What's the weather in Lagos?")
    print(json.dumps({k: r[k] for k in ("type", "function_calls", "confidence", "peak_ram_mb")}))

    # Full loop: Cactus runs your function and returns its result
    agent.reset()
    print(agent.run("What's the weather in Lagos?")["results"])

Run it with python hello.py. The first run downloads the engine and the 35 MB weights (about 10 seconds for us); later runs load from the cache in well under a second. Our output:

{"type": "call", "function_calls": [{"name": "get_weather", "arguments": {"city": "Lagos"}}], "confidence": 0.9555, "peak_ram_mb": 134.2}
[{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

What happened:

  • @cactus.tool turns the function signature and docstring into a JSON schema. Type hints become argument types; a default makes an argument optional [2].
  • complete() runs one turn and returns the proposed call without executing anything. Use it when you want to check or confirm before acting.
  • run() executes the loop: the model picks calls, Cactus runs your functions, feeds results back and returns them under results [2].
  • reset() clears the conversation but keeps the tools. One toolset per agent: to change tools, create a new Cactus [2].

5. Step 3: design tools a small model can use

A 35 MB model reads your whole toolset as part of its prompt and has no retrieval step, so tool design drives accuracy. Cactus recommends five tools or fewer, plain names without shared prefixes, closed choices as enums and bounded numbers [2]. The engine compiles your schemas into a grammar, so every argument it returns matches the schema; tighter schemas therefore leave less room to be wrong.

  • Use typing.Literal[...] for closed sets (rooms, priorities, categories).
  • Bound numbers with Annotated[int, cactus.Field(ge=0, le=100)] [2].
  • Write the docstring as one plain sentence, with Args: lines for anything ambiguous.
  • Pass facts such as the date and locale through system=, for example "date: 2026-11-16 Mon 10:00; locale: en-GB", so "tomorrow at 7" can resolve [2].

6. Step 4: confidence gating and a fallback

Every response carries a confidence score between 0 and 1. The engine already moves calls below 0.1 into suppressed_calls; above that, the threshold is your product decision [2]. The script below acts at or above 0.7, asks for confirmation below it, and sends refusals to an OpenAI-compatible chat endpoint you control (a vLLM or llama.cpp server on your own GPU box, or a cloud API where the data allows). With no endpoint configured, it hands the request to a person.

import json
import os
import urllib.request
from typing import Literal

import cactus

CONFIDENCE_FLOOR = 0.7  # tune per product on your own test set


@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
    """Set a room's light brightness.

    Args:
        room: which room
        brightness: brightness in percent, 0 to 100
    """
    return {"room": room, "brightness": brightness}


@cactus.tool
def create_ticket(summary: str, priority: Literal["low", "normal", "urgent"] = "normal"):
    """Open a support ticket.

    Args:
        summary: one-line description of the problem
    """
    return {"ticket": "T-1001", "summary": summary, "priority": priority}


TOOLS = {"set_lights": set_lights, "create_ticket": create_ticket}


def cloud_fallback(text: str) -> dict:
    """Send the request to an OpenAI-compatible endpoint you control, if configured."""
    url = os.environ.get("FALLBACK_URL")  # e.g. http://gpu-box:8000/v1/chat/completions
    if not url:
        return {"route": "human", "note": "no fallback configured; ask the user to confirm"}
    body = json.dumps({
        "model": os.environ.get("FALLBACK_MODEL", "default"),
        "messages": [{"role": "user", "content": text}],
    }).encode()
    req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        answer = json.load(resp)["choices"][0]["message"]["content"]
    return {"route": "cloud", "answer": answer}


def handle(agent, text: str) -> dict:
    agent.reset()
    r = agent.complete(text)
    calls, held = r["function_calls"], r.get("suppressed_calls") or []
    conf = r["confidence"]
    if calls and conf is not None and conf >= CONFIDENCE_FLOOR:
        results = [TOOLS[c["name"]](**c["arguments"]) for c in calls]
        return {"route": "local", "confidence": conf, "results": results}
    if calls or held:
        return {"route": "confirm", "confidence": conf, "proposed": calls or held}
    return cloud_fallback(text)


if __name__ == "__main__":
    with cactus.Cactus(tools=list(TOOLS.values()),
                       system="date: 2026-11-16 Mon 10:00; locale: en-GB") as agent:
        for text in [
            "dim the kitchen to 20 percent",
            "the printer on floor 2 is on fire, raise it as urgent",
            "write me a haiku about invoices",
        ]:
            print(text, "->", json.dumps(handle(agent, text)))

Our output, with no fallback configured:

dim the kitchen to 20 percent -> {"route": "local", "confidence": 0.9622, "results": [{"room": "kitchen", "brightness": 20}]}
the printer on floor 2 is on fire, raise it as urgent -> {"route": "confirm", "confidence": 0.3689, "proposed": [{"name": "set_lights", "arguments": {"room": "lounge", "brightness": 100}}]}
write me a haiku about invoices -> {"route": "human", "note": "no fallback configured; ask the user to confirm"}

The second line is the reason for the gate. The model picked the wrong tool entirely, but with low confidence, so the user would have seen a confirmation instead of the lounge lights coming on. The third request was correctly refused. Set FALLBACK_URL (for example http://gpu-box:8000/v1/chat/completions) and FALLBACK_MODEL to route refusals to a larger model.

Routing a Cactus response: calls with confidence at or above the threshold run locally; calls below it or suppressed calls go to user confirmation; no calls escalate to a larger model or a person.
Three outcomes, decided from fields the engine already returns.

7. Step 5: structured extraction that refuses invented values

cactus.extract() declares your record as the only tool and returns the fields. In strict mode (the default) it raises ExtractionValidationError if a value is not grounded in the text [2].

import cactus

INVOICE = {
    "name": "invoice",
    "description": "An invoice record",
    "parameters": {
        "type": "object",
        "properties": {
            "vendor": {"type": "string"},
            "total": {"type": "number"},
            "currency": {"type": "string"},
        },
        "required": ["vendor", "total"],
    },
}

for text in [
    "Invoice from Acme Ltd. Amount due: 1200 GBP.",
    "Invoice from Acme Ltd, total GBP 1,200.00, due 30 November.",
]:
    try:
        print("ok:", cactus.extract(text, INVOICE))
    except cactus.ExtractionValidationError as err:
        print("rejected:", err)
        print("  unchecked value would have been:", cactus.extract(text, INVOICE, strict=False))
ok: {'vendor': 'Acme Ltd. Amount', 'total': 1200.0}
rejected: extraction returned values not grounded in the input: total
  unchecked value would have been: {'vendor': 'Acme Ltd', 'total': 12200.0, 'currency': 'GBP'}

Two lessons from one small test. Strict mode caught a wrong total (12,200 instead of 1,200) that would otherwise have gone into your system. And the first result passed strict mode with a messy vendor name ("Acme Ltd. Amount"), because grounding checks that a value comes from the text, not that it is the right span. Keep your own validation for anything that matters.

8. Step 6: speech

Whistle needs 16 kHz mono WAV, or 16 kHz float samples. Make a test clip:

# macOS: synthesise a test clip and convert it to 16 kHz mono WAV
say -o lights.aiff "Turn the office lights down to forty percent."
afconvert -f WAVE -d LEI16@16000 -c 1 lights.aiff lights.wav

# Linux or Windows: convert any recording with ffmpeg
ffmpeg -i recording.m4a -ar 16000 -ac 1 lights.wav
import json
from typing import Literal

import cactus


@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
    """Set a room's light brightness.

    Args:
        room: which room
        brightness: brightness in percent, 0 to 100
    """
    return {"room": room, "brightness": brightness}


# 1. Plain transcription with word timestamps
t = cactus.transcribe("lights.wav", language="en", word_timestamps=True)
print(t["text"], "| ttft_ms:", t["ttft_ms"])
print([(w["word"] if "word" in w else w, w.get("start"), w.get("end")) for w in t["words"][:3]])

# 2. Speech straight to a tool call (no separate transcription step)
with cactus.Cactus(tools=[set_lights]) as agent:
    r = agent.complete(audio="lights.wav", language="en")
    print(r["audio_text"], "->", json.dumps(r["function_calls"]), r["confidence"])
Turn the office lights down to 40%. | ttft_ms: 298.1
[('Turn', 0.0, 0.24), ('the', 0.24, 0.4), ('office', 0.4, 0.64)]
Turn the office lights down to 40%. -> [{"name": "set_lights", "arguments": {"room": "office", "brightness": 40}}] 0.9515

Passing audio= to complete() goes from speech to a tool call in one step and returns the transcript as audio_text [2]. For a 5-second clip, transcription took 0.74 seconds once loaded. For live microphone input, cactus.stream() takes roughly one-second chunks and yields committed text as it goes; install cactus-compute[mic] for microphone capture and resampling [2].

Limits worth knowing: seven languages (English, German, French, Spanish, Italian, Dutch, Polish); silence returns empty text, but music and other noise can produce words, so put a voice activity detector or wake word in front of an always-on microphone [2]. Cactus reports Whistle beating Whisper base on several word error rate test sets and losing on others, at 16.9 MB against 145.3 MB [6]. We did not reproduce those benchmarks; our test clips were synthetic speech, which is easier than a real room.

9. Step 7: vision

import json

from PIL import Image, ImageDraw, ImageFont

from cactus import Mirage

# Make a test image so the example is self-contained
img = Image.new("RGB", (640, 400), "white")
draw = ImageDraw.Draw(img)
font = ImageFont.load_default(size=40)
draw.text((40, 40), "INVOICE 1042", fill="black", font=font)
draw.text((40, 120), "Total: GBP 1,200.00", fill="black", font=font)
draw.ellipse((420, 220, 560, 360), fill="red")
img.save("card.png")

vision = Mirage()

# Text lines with confidence (boxes are on a 0-1000 grid)
for line in vision.describe("card.png", sections=["text"])["text"]:
    print(f'{line["score"]:.2f}  {line["string"]}  {line["bbox"]}')

# Fill a small form with closed choices
form = {
    "document_type": {"enum": ["invoice", "receipt", "letter", "other"]},
    "has_red_circle": {"enum": ["yes", "no"]},
}
for field, answer in vision.extract("card.png", form)["form"].items():
    print(field, "->", answer["value"])
0.38  INVOICE  [66, 113, 321, 210]
0.70  INVOICE 1042  [60, 108, 464, 216]
0.50  1042  [321, 109, 472, 216]
0.37  Total: GBP  [55, 304, 369, 420]
0.67  Total: GBP 1,200.00  [52, 303, 620, 423]
0.51  1,200.00  [381, 310, 612, 418]
document_type -> invoice
has_red_circle -> None

Text reading worked, with overlapping candidates at different confidence levels; keep the highest-scoring line per region. Boxes come back on a 0 to 1,000 grid, not in pixels (our 640 by 400 image's red circle came back near x 650 to 880). The form answered one question and abstained on the other (None, with the probabilities in the full result). In an earlier run with a different font it answered the red-circle question wrongly. Mirage is useful for reading text and rough layout; for decisions, validate on your own images and gate on the returned probabilities.

10. Step 8: measure latency and memory

import os
import statistics
import time

import cactus

cactus.set_threads(int(os.environ.get("CACTUS_THREADS", "1")))


@cactus.tool
def set_lights(room: str, brightness: int):
    "Set a room's light brightness in percent."
    return {"room": room, "brightness": brightness}


QUERIES = ["dim the kitchen to 30 percent", "office lights to 80", "turn the lounge down to 10"]

t0 = time.perf_counter()
agent = cactus.Cactus(tools=[set_lights])
print(f"load: {time.perf_counter() - t0:.2f}s")

agent.complete(QUERIES[0])  # warm-up, not counted
wall, decode, ram = [], [], []
for i in range(30):
    agent.reset()
    t = time.perf_counter()
    r = agent.complete(QUERIES[i % len(QUERIES)])
    wall.append((time.perf_counter() - t) * 1000)
    decode.append(r["decode_tps"])
    ram.append(r["peak_ram_mb"])
agent.close()

wall.sort()
print(f"p50 {statistics.median(wall):.0f} ms  p95 {wall[int(0.95 * len(wall)) - 1]:.0f} ms")
print(f"decode median {statistics.median(decode):.0f} tok/s  peak RAM {max(ram):.0f} MB")

Run it with CACTUS_THREADS=1 python bench.py, then with 0 (all cores) and 4. Two of our runs:

CACTUS_THREADS=1: p50 105 ms  p95 112 ms   decode median 451 tok/s  peak RAM 136 MB
CACTUS_THREADS=0: p50 81 ms   p95 167 ms   decode median 534 tok/s  peak RAM 136 MB
Latency by thread count. Moderate load: all cores 81 ms, one thread 105 ms, four threads 129 ms. Saturated machine: one thread 110 ms, all cores 1,392 ms, four threads 1,475 ms with a p95 of 21.8 seconds.
On a quiet machine more threads helped a little; on a saturated one they made latency far worse. One thread stayed steady throughout.

Our Mac was shared with other heavy jobs, which proved useful. When other work saturated the cores, the default setting pushed the median to 1.4 seconds and four threads gave a 22-second p95, while one thread stayed at about 110 ms. A phone that is also rendering UI can hit the same contention. Start with one or two threads and measure on the slowest device you support. peak_ram_mb in each response is the engine's own figure; the whole Python process peaked at about 176 to 197 MB in our runs.

11. Step 9: package it to run offline

Nothing should download on a customer's device or in production. Fetch the engine and weights at build time and point the package at them [3].

A container that runs with networking off

FROM python:3.12-slim

ENV CACTUS_TELEMETRY=0 \
    XDG_CACHE_HOME=/opt/cache \
    PIP_NO_CACHE_DIR=1

RUN pip install "cactus-compute==4.2.2"

# Fetch the engine and the weights at build time; nothing downloads at run time.
RUN cactus fetch \
 && cactus download needle3 --out /opt/models \
 && chmod -R a+rX /opt/cache /opt/models

ENV HF_HUB_OFFLINE=1 \
    CACTUS_NEEDLE_WEIGHTS=/opt/models/needle3.cact

RUN useradd --create-home app
USER app
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]
docker build -t cactus-edge-demo .
docker run --rm --network none cactus-edge-demo

We built this on Apple silicon (so a Linux arm64 image of about 134 MB) with gated.py as app.py, and it printed the same three routes as step 4 with networking disabled. One trap we hit: cactus download needle3 alone writes into the current directory, so without --out and CACTUS_NEEDLE_WEIGHTS the non-root user tried to download the weights again at run time and failed on a read-only cache.

Build time with internet: pip install a pinned version, cactus fetch, cactus download needle3. Run time with no network: HF_HUB_OFFLINE=1, CACTUS_TELEMETRY=0 and CACTUS_NEEDLE_WEIGHTS pointing at the bundled file.
Fetch at build time, run with the network off.

An air-gapped Linux device

# On a connected machine with the same OS, CPU architecture and Python as the device
pip download "cactus-compute==4.2.2" -d wheels
cactus fetch --platform-tag manylinux2014_aarch64 --out ./engine   # Linux Arm64 engine
cactus download needle3 --out ./models

# Copy wheels/, engine/ and models/ to the device, then on the device:
pip install --no-index --find-links wheels cactus-compute
export HF_HUB_OFFLINE=1 CACTUS_TELEMETRY=0
export CACTUS_LIB_PATH=$PWD/engine/libcactus.so
export CACTUS_NEEDLE_WEIGHTS=$PWD/models/needle3.cact

We checked the equivalent on macOS with an empty cache directory, HF_HUB_OFFLINE=1 and the two environment variables set: the tool call ran without touching the network. Use CACTUS_WHISTLE_WEIGHTS and CACTUS_MIRAGE_WEIGHTS for the other models [3].

Mobile and browser apps

For apps outside Python, cactus download <platform> --model needle3 fetches a native build: iOS, tvOS and watchOS get a static library and a C header (cactus.h); Android and Linux also get a command-line runner; the browser target is WebAssembly [5]. We did not build a mobile app for this note; follow Cactus's deployment guide and treat it as a C library you wrap in Swift or Kotlin.

12. Test and evaluate before you ship

The only accuracy figures that matter are the ones on your requests. Write a test file of real queries with the expected call, including ones that should be refused, then score it at several thresholds:

{"query": "dim the kitchen to 20 percent", "expect": {"name": "set_lights", "arguments": {"room": "kitchen", "brightness": 20}}}
{"query": "the wifi in the office keeps dropping", "expect": {"name": "create_ticket"}}
{"query": "what's the capital of France?", "expect": null}
import json
import sys

import cactus

from gated import TOOLS  # reuse the tool definitions


def matches(call, expect):
    if expect is None:
        return call is None
    if call is None or call["name"] != expect["name"]:
        return False
    wanted = expect.get("arguments", {})
    return all(call["arguments"].get(k) == v for k, v in wanted.items())


cases = [json.loads(line) for line in open(sys.argv[1])]
rows = []
with cactus.Cactus(tools=list(TOOLS.values()), system="locale: en-GB") as agent:
    for case in cases:
        agent.reset()
        r = agent.complete(case["query"])
        call = r["function_calls"][0] if r["function_calls"] else None
        rows.append((r["confidence"] or 0.0, call is not None, matches(call, case["expect"]), case))

for threshold in (0.0, 0.5, 0.7, 0.9):
    acted = [row for row in rows if row[1] and row[0] >= threshold]
    wrong = [row for row in acted if not row[2]]
    missed = sum(1 for row in rows if row[3]["expect"] is not None and row not in acted)
    print(f"threshold {threshold:.1f}: acted on {len(acted)}/{len(rows)}, "
          f"wrong actions {len(wrong)}, not acted (sent to confirm/fallback) {missed}")
for conf, has_call, ok, case in rows:
    print(f"{conf:.2f} {'ok ' if ok else 'BAD'} {case['query']}")

On our seven-case file (save it as cases.jsonl beside gated.py and run python evaluate.py cases.jsonl):

threshold 0.0: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.5: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.7: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
threshold 0.9: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
0.95 ok  dim the kitchen to 20 percent
0.97 ok  office lights to 75
0.96 ok  make the lounge brighter, 90 percent
0.22 BAD the wifi in the office keeps dropping
0.62 ok  log an urgent ticket: server room is overheating
0.96 ok  what's the capital of France?
0.90 ok  write me a poem

Seven cases prove nothing statistically; 100 to 200 real ones per tool surface is a sensible start. The shape is what matters: raising the threshold trades actions for confirmations. Pick the point where wrong actions are rare enough for the task, and rerun the file in CI whenever you change the package version, weights or tool definitions. Cactus ships six ready-made tool surfaces with acceptance suites in cactus.environments, and notes that the base model does not pass five of them at the default gate [2], a useful reminder that a small model's fit depends on the tools.

If the base model is not accurate enough on your tools, Cactus supports LoRA fine-tuning (a cheap way to adapt a model by training small extra weight matrices) locally with cactus-compute[train], or on its paid hosted platform, producing a new .cact you load with weights= [2]. Note that locally fine-tuned weights do not include the calibration head, so confidence comes back as None [2]; you would need another gate.

13. Running it in production

  • Pin everything. Package version, and the weights file you tested with. Weekly releases are normal for this project.
  • Threads. Set cactus.set_threads() explicitly, before loading models; start at one or two on shared or mobile CPUs.
  • Concurrency. Text, speech and vision share one runtime and calls are serialised across threads [3]. For a server handling many requests, run several worker processes rather than threads.
  • Security. The model chooses which of your functions to call. Only expose functions that are safe to run with any schema-valid arguments, and re-check arguments and permissions in the function itself.
  • Monitoring. Log route (local, confirm, fallback), confidence, latency, peak_ram_mb and whether the user corrected the action. Avoid logging transcripts or extracted personal data unless you need them and have a basis to keep them.
  • Privacy. Disable telemetry where your policy requires it, and remember that the fallback path sends data off the device: apply the same data rules there.

14. Pitfalls and debugging

  • Empty function_calls everywhere. The request does not map to a tool, or the tool description is unclear. Check the reasoning field and rewrite the docstring.
  • confidence is None. You loaded locally fine-tuned weights without the calibration head [2].
  • ExtractionValidationError. A value was not found in the text. Do not switch to strict=False to make it go away; inspect it.
  • PermissionError on start in a container. The package is trying to download into a cache it cannot write. Bundle the files and set CACTUS_NEEDLE_WEIGHTS and HF_HUB_OFFLINE=1.
  • Latency spikes. CPU contention with multi-threading. Try set_threads(1).
  • Wrong sample rate. Whistle expects 16 kHz mono; convert, or install the [mic] extra for resampling [1].

15. How it compares with the alternatives

OptionWhat you runTypical sizeBest forTrade-offs
Cactus (cactus-compute)Cactus's own Needle, Whistle and Mirage models on one engine17 to 35 MB per modelNarrow tool calls, extraction, short speech, OCR on any CPU, offlineNo free text; small tool sets; young project; few independent benchmarks
llama.cpp (and Ollama on top)Open LLMs and some vision models in GGUF format [11]Hundreds of MB to tens of GBGeneral chat, drafting and tool use with 1B to 70B+ models on laptops and serversFar more memory; tool-calling reliability varies by model
MLC LLMOpen LLMs compiled for phones, browsers and GPUs [14]Hundreds of MB and upChat-class models inside mobile and web appsCompilation step per model and target
ONNX RuntimeYour own model exported to ONNX [12]Whatever you exportShipping a custom-trained model across platforms and acceleratorsYou own training, export and pre/post-processing
ExecuTorchYour PyTorch model, exported for edge devices; 1.0 since October 2025 [13]Whatever you exportPyTorch teams targeting phones and embedded NPUsMore engineering; you bring the model
NVIDIA TensorRT Edge-LLMLLMs, vision-language and speech models on Jetson, DRIVE and DGX Spark [10]GB-class modelsRobots, vehicles and edge boxes with NVIDIA GPUs; NVIDIA's own Nemotron modelsNVIDIA hardware only; engines built per device

NVIDIA's own on-device direction is larger models on NVIDIA edge hardware: its Nemotron 3 Nano Omni, for example, is a 30B-parameter mixture-of-experts model with about 3B active per token, aimed at Jetson and DGX Spark among other targets [15]. That is a different class from a 35 MB CPU model. A sensible architecture often uses both: a tiny model on the user's device for the frequent, narrow requests, and a larger model on a GPU server you control for the fallback.

16. FAQ

Is this an NVIDIA product? No. Cactus Compute builds and publishes it under Apache 2.0, including the prebuilt runtime binaries [1]. The nearest NVIDIA link is that Cactus also ships a Mac runner, cactus-mlx, for compressed checkpoints of NVIDIA's Nemotron models [1].

Can Needle answer questions or summarise text? No. It returns tool calls or extracted fields only. Use a larger model for prose.

Which languages does speech support? English, German, French, Spanish, Italian, Dutch and Polish; fine-tuning can add others [2].

Can I use it commercially? The code and runtime binaries are Apache 2.0 [1], and the Mirage model card lists Apache 2.0 [7]. Check the licence of each model you ship on its Hugging Face card.

If you are deciding which AI work should run on devices, on your own GPU servers or in the cloud, and need the private side built and run, see how we approach private AI.

References

  1. cactus-compute/cactus: README, GitHub, Cactus Compute.
  2. Cactus API reference (llms.txt), Cactus Compute.
  3. Needle Python API and CLI documentation, Cactus Compute.
  4. Needle supported devices, Cactus Compute.
  5. Cactus runtime binaries and platforms, Hugging Face.
  6. Whistle announcement, Cactus Compute, 2 October 2026.
  7. Mirage model card, Hugging Face.
  8. cactus-compute on PyPI (release history).
  9. Cactus Compute company page, Y Combinator.
  10. TensorRT Edge-LLM overview, NVIDIA.
  11. llama.cpp, GitHub.
  12. ONNX Runtime, Microsoft.
  13. Introducing ExecuTorch 1.0, PyTorch.
  14. MLC LLM, GitHub.
  15. NVIDIA Nemotron 3 Nano Omni, NVIDIA blog, 28 April 2026.

All code in this note was run by Workstation on 16 November 2026 with cactus-compute 4.2.2 on an Apple M4 Max and in a Linux arm64 container; outputs are copied from those runs. Performance figures are indicative, from a shared machine. Other figures are vendor claims. This note is independent and not endorsed by Cactus Compute or NVIDIA. NVIDIA, Jetson, DRIVE, DGX, Nemotron and TensorRT are trademarks of NVIDIA Corporation. Other names are the property of their respective owners.

分享這篇文章

更多 Technology 內容

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Evidence instead of proofs, in about 400 lines of Python with real output: slice metrics with bootstrap intervals and a pass/fail contract, Hypothesis property and metamorphic tests, calibration and conformal prediction, Pydantic contracts and regression evals for LLM output, drift checks, a CI gate, and exact verification of a small neural network

閱讀更多
TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

Install torchcurves, fit and size a curve, choose B-spline or Legendre and the input map, build a monotone calibration curve and an additive model on tabular data, compare with SciPy, scikit-learn, a KAN and an MLP, export coefficients to JSON, test and run it in production, with real outputs

閱讀更多
AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

Close the AI guidance gap in code: a versioned policy file, an OpenAI-compatible FastAPI gateway, Presidio redaction, per-team tool and MCP allow-lists, red-team tests, an audit log, Prometheus metrics and a usage report

閱讀更多