ร้านเปิดให้สั่งแล้ว

เราสร้างเครื่อง Workstation AI สำหรับเว็บไซต์ เอเจนต์ AI และการเทรน LLM — กำหนดค่าและซื้อออนไลน์

ดูราคาสำหรับคู่ค้า
ผลิตภัณฑ์ AI
AI LabsOpenAI AgentsClaude AgentsGrok BotCRM พื้นฐานของ Workstation AIการตลาดผลิตภัณฑ์ทั้งหมด
โซลูชัน AI
AI ประยุกต์เอเจนต์ AIAI เชิงสร้างระบบอัตโนมัติด้วย AIยกระดับทักษะ AIแพ็กเกจ AI สำหรับ SMEห้องปฏิบัติการ AI สำหรับองค์กรเวิร์กสเตชัน AIฮาร์ดแวร์และซอฟต์แวร์ NVIDIANVIDIA Accelerated ComputingAI ส่วนตัวGPU ไพรเวทคลาวด์คลัสเตอร์ GPUEdge AIAI ตามอุตสาหกรรม
บริการ AI
ปรับปรุงแพลตฟอร์มให้ทันสมัยวิศวกรรมดิจิทัลรากฐานข้อมูลและ AIการปฏิบัติงานอัตโนมัติที่ปรึกษา AIระบบอัตโนมัติ DevOpsความมั่นคงปลอดภัยไซเบอร์การพัฒนาซอฟต์แวร์การสร้างเอเจนต์การตั้งค่า MLOpsการเช่า GPUGPU ไพรเวทคลาวด์
เกี่ยวกับเรา
บริษัทเรื่องราวของเราพันธมิตรเรื่องราวของลูกค้า
ศูนย์ข้อมูล AI
ศูนย์ข้อมูล AIGPU ไพรเวทคลาวด์คลัสเตอร์ GPUการเช่า GPU
ข้อมูลเชิงลึก AIชุด Jevแล็บเบนช์มาร์กการเลือกโมเดล
บันทึกทางเทคนิค
เอกสารประกอบ
ระบบปฏิบัติการ LinuxUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
แพลตฟอร์มคลาวด์ AIAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI Foundryคลาวด์และมาร์เก็ตเพลส GPU
ศูนย์ข้อมูล AI ส่วนตัวสำหรับสำนักงานกฎหมายสำหรับสำนักงานบัญชีแพลตฟอร์มภายในองค์กรคลาวด์ที่ปลอดภัยเต็มรูปแบบ
เครื่องมือRing PromoterJobshout
ข่าวสาร
บล็อก
ติดต่อเราลงชื่อเข้าใช้

เวิร์กสเตชัน AI ซอฟต์แวร์มัลติเอเจนต์ AI โครงสร้างพื้นฐาน GPU และโซลูชันเอเจนต์อัจฉริยะสำหรับธุรกิจยุคใหม่

ติดต่อเรา

โซลูชัน AI

AI ประยุกต์เอเจนต์ AIAI เชิงสร้างระบบอัตโนมัติด้วย AIยกระดับทักษะ AIแพ็กเกจ AI สำหรับ SMEห้องปฏิบัติการ AI สำหรับองค์กรAI ตามอุตสาหกรรม

ผลิตภัณฑ์

ผลิตภัณฑ์ทั้งหมดWorkstation AI native CRM และ ERPการตลาดOpenAI AgentsRing PromoterJobshout

บริษัท

เกี่ยวกับเราบริษัทศูนย์ข้อมูล AIทำไมต้อง Workstationพันธมิตรเรื่องราวของลูกค้าราคาติดต่อTenthMatrix

แหล่งข้อมูล

ข้อมูลเชิงลึก AIบันทึกทางเทคนิคเอกสารประกอบระบบปฏิบัติการ Linuxแพลตฟอร์มคลาวด์ AIศูนย์ข้อมูล AI ส่วนตัวบล็อกค้นหาแผนผังเว็บไซต์
สำนักงานใหญ่ สหราชอาณาจักร
77-79 Marlowes, Hemel Hempstead HP1 1LFเส้นทาง - ออกทางแยกที่ 20 จาก M25 Outer Londonเลขทะเบียนบริษัท: 11641870จ. - ศ.: 9:00 - 18:00 น. GMT
+44 7515 356 146
สำนักงานเบลเยียม
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683จ. - ศ.: 9:00 - 18:00 น. CET
+32 492 45 67 46
สำนักงานอินเดีย
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
สำนักงานไต้หวัน
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓จันทร์ - ศุกร์: 9:00 - 18:00 น. เวลาไต้หวัน (GMT+8)
+886 926 892 701+32 492 45 67 46

© 2026 Workstation AI สงวนลิขสิทธิ์

ความเป็นส่วนตัวการลบข้อมูลคุกกี้ข้อกำหนดในการให้บริการแผนผังเว็บไซต์

NVIDIA, the NVIDIA logo, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX and HGX are trademarks and/or registered trademarks of NVIDIA Corporation in the US and other countries. AMD, the AMD Arrow logo, EPYC, Instinct, Ryzen and Threadripper are trademarks of Advanced Micro Devices, Inc. Dell and PowerEdge are trademarks of Dell Inc. or its subsidiaries. Intel and Xeon are trademarks of Intel Corporation or its subsidiaries. AMAX, AceleMax and LiquidMax are trademarks of AMAX Engineering Corporation. All other product names, logos and brands are the property of their respective owners. We use them only to identify the products we describe, supply or support; their use does not mean their owners endorse us.

หน้าแรก / บันทึกทางเทคนิค / Technology
AIAI AgentsPythonLLM

Run Agentic, Voice and Vision Models On-Device with Cactus: A Hands-On Guide (2026)

Install cactus-compute, make tool calls, gate on confidence with a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, and ship an offline container, with every example run and a comparison with llama.cpp, ONNX Runtime, ExecuTorch, MLC LLM and TensorRT Edge-LLM

16 พฤศจิกายน 2569Technology22 min read

A hands-on guide for AI developers who want to run small agentic, speech and vision models on the device today. We install Cactus Compute's open-source engine, make a tool call, add confidence gating and a fallback to a bigger model, extract structured data, run speech and vision, measure latency and memory, then package it all to run with the network switched off. Every example below was run on our own hardware, and we show what it printed, including where the models got things wrong.

A terminal running pip install cactus-compute and a JSON tool call, connected to a laptop, a phone and a single-board computer running the same model offline.

Summary. pip install cactus-compute gives you three small models on one native engine: Needle 3 (35.3 MB) turns a request into a call to one of your Python functions or extracts fields from text; Whistle (16.9 MB) transcribes speech in seven European languages; Mirage (21.6 MB) returns objects, text and simple form answers from an image. They run on a CPU, offline after the first download. In our runs a tool call took about 70 to 110 ms with 136 MB peak memory. The models are narrow and sometimes wrong, so production use rests on three habits: strict schemas, a confidence threshold chosen from your own test set, and an escalation path to a larger model or a person.

This technote accompanies our blog post Sub-50 MB on-device AI: what Cactus's engine means for business, which covers the business case. One correction first: this engine is sometimes reported as an NVIDIA product. It is not. It is built by Cactus Compute, a Y Combinator company, and released under Apache 2.0 [1][9]. Facts and versions are as of 16 November 2026.

1. Who this is for, what you will build, what you need

Who it is for. Python developers and ML engineers who want AI features that work without a network round trip: voice commands in a field app, a kiosk that reads documents, a support tool that turns a message into a ticket, or a device that must keep working offline.

What you will build. A first tool call; a confidence-gated agent that acts, asks or escalates to an OpenAI-compatible endpoint; extraction that refuses invented values; speech to a tool call; image text reading; a latency benchmark and an evaluation harness; and an offline container.

Prerequisites. Python 3.9 or later [1]; macOS on Apple silicon, Linux (x86-64 or Arm64) or Windows, among the platforms Cactus lists [4]; about 200 MB of free disk for the package, engine and all three models; internet access for the first download only. No GPU is needed. We used an Apple M4 Max with macOS 26.7, Python 3.15 and cactus-compute 4.2.2 (engine 4.1.1), plus Docker for the Linux arm64 container.

2. How the pieces fit

The Python package is a thin layer: it turns your functions into JSON schemas, runs the agent loop, checks that arguments are grounded in the input and reports a confidence score. It calls one native engine library, which loads a separate weights file (.cact) per model [2]. On first use the engine and weights are downloaded from Hugging Face at pinned commits, checked against SHA-256 checksums and cached under ~/.cache/cactus [2][3].

Architecture: your Python app, the cactus Python layer, the native libcactus engine and three weights files, with a first-use download from Hugging Face into a local cache.
One engine, three weights files, a thin Python layer. Sizes are the files we measured.

Two things to know before you start. First, Needle 3 does not write prose. It returns a tool call, or an empty list when the request is off topic, which is a refusal [2]. If you need written answers, you need a bigger model (see the comparison in section 13). Second, Cactus describes the engine as "sub-1 MB" [2]; the uncompressed libraries we downloaded were 1.4 MB on macOS and 1.8 MB on Linux Arm64. Each model plus the engine is still well under 50 MB.

3. Step 1: install

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install "cactus-compute==4.2.2"

# Optional: switch off anonymous usage counts for this shell
export CACTUS_TELEMETRY=0

The pin matters: the package went from 4.0.0 to 4.2.3 in eleven days in November 2026 [8]. import cactus does not pull in JAX; JAX is only needed for the optional fine-tuning extra, cactus-compute[train] [2]. The package sends anonymous usage counts (function name, version, OS and a random install id) unless CACTUS_TELEMETRY=0 or DO_NOT_TRACK=1 is set [1][2].

4. Step 2: your first tool call

This is the README example [1], extended to show both ways of driving the agent.

import json

import cactus


@cactus.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}


with cactus.Cactus(tools=[get_weather]) as agent:
    # One turn: the model proposes a call, you decide whether to run it
    r = agent.complete("What's the weather in Lagos?")
    print(json.dumps({k: r[k] for k in ("type", "function_calls", "confidence", "peak_ram_mb")}))

    # Full loop: Cactus runs your function and returns its result
    agent.reset()
    print(agent.run("What's the weather in Lagos?")["results"])

Run it with python hello.py. The first run downloads the engine and the 35 MB weights (about 10 seconds for us); later runs load from the cache in well under a second. Our output:

{"type": "call", "function_calls": [{"name": "get_weather", "arguments": {"city": "Lagos"}}], "confidence": 0.9555, "peak_ram_mb": 134.2}
[{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

What happened:

  • @cactus.tool turns the function signature and docstring into a JSON schema. Type hints become argument types; a default makes an argument optional [2].
  • complete() runs one turn and returns the proposed call without executing anything. Use it when you want to check or confirm before acting.
  • run() executes the loop: the model picks calls, Cactus runs your functions, feeds results back and returns them under results [2].
  • reset() clears the conversation but keeps the tools. One toolset per agent: to change tools, create a new Cactus [2].

5. Step 3: design tools a small model can use

A 35 MB model reads your whole toolset as part of its prompt and has no retrieval step, so tool design drives accuracy. Cactus recommends five tools or fewer, plain names without shared prefixes, closed choices as enums and bounded numbers [2]. The engine compiles your schemas into a grammar, so every argument it returns matches the schema; tighter schemas therefore leave less room to be wrong.

  • Use typing.Literal[...] for closed sets (rooms, priorities, categories).
  • Bound numbers with Annotated[int, cactus.Field(ge=0, le=100)] [2].
  • Write the docstring as one plain sentence, with Args: lines for anything ambiguous.
  • Pass facts such as the date and locale through system=, for example "date: 2026-11-16 Mon 10:00; locale: en-GB", so "tomorrow at 7" can resolve [2].

6. Step 4: confidence gating and a fallback

Every response carries a confidence score between 0 and 1. The engine already moves calls below 0.1 into suppressed_calls; above that, the threshold is your product decision [2]. The script below acts at or above 0.7, asks for confirmation below it, and sends refusals to an OpenAI-compatible chat endpoint you control (a vLLM or llama.cpp server on your own GPU box, or a cloud API where the data allows). With no endpoint configured, it hands the request to a person.

import json
import os
import urllib.request
from typing import Literal

import cactus

CONFIDENCE_FLOOR = 0.7  # tune per product on your own test set


@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
    """Set a room's light brightness.

    Args:
        room: which room
        brightness: brightness in percent, 0 to 100
    """
    return {"room": room, "brightness": brightness}


@cactus.tool
def create_ticket(summary: str, priority: Literal["low", "normal", "urgent"] = "normal"):
    """Open a support ticket.

    Args:
        summary: one-line description of the problem
    """
    return {"ticket": "T-1001", "summary": summary, "priority": priority}


TOOLS = {"set_lights": set_lights, "create_ticket": create_ticket}


def cloud_fallback(text: str) -> dict:
    """Send the request to an OpenAI-compatible endpoint you control, if configured."""
    url = os.environ.get("FALLBACK_URL")  # e.g. http://gpu-box:8000/v1/chat/completions
    if not url:
        return {"route": "human", "note": "no fallback configured; ask the user to confirm"}
    body = json.dumps({
        "model": os.environ.get("FALLBACK_MODEL", "default"),
        "messages": [{"role": "user", "content": text}],
    }).encode()
    req = urllib.request.Request(url, body, {"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        answer = json.load(resp)["choices"][0]["message"]["content"]
    return {"route": "cloud", "answer": answer}


def handle(agent, text: str) -> dict:
    agent.reset()
    r = agent.complete(text)
    calls, held = r["function_calls"], r.get("suppressed_calls") or []
    conf = r["confidence"]
    if calls and conf is not None and conf >= CONFIDENCE_FLOOR:
        results = [TOOLS[c["name"]](**c["arguments"]) for c in calls]
        return {"route": "local", "confidence": conf, "results": results}
    if calls or held:
        return {"route": "confirm", "confidence": conf, "proposed": calls or held}
    return cloud_fallback(text)


if __name__ == "__main__":
    with cactus.Cactus(tools=list(TOOLS.values()),
                       system="date: 2026-11-16 Mon 10:00; locale: en-GB") as agent:
        for text in [
            "dim the kitchen to 20 percent",
            "the printer on floor 2 is on fire, raise it as urgent",
            "write me a haiku about invoices",
        ]:
            print(text, "->", json.dumps(handle(agent, text)))

Our output, with no fallback configured:

dim the kitchen to 20 percent -> {"route": "local", "confidence": 0.9622, "results": [{"room": "kitchen", "brightness": 20}]}
the printer on floor 2 is on fire, raise it as urgent -> {"route": "confirm", "confidence": 0.3689, "proposed": [{"name": "set_lights", "arguments": {"room": "lounge", "brightness": 100}}]}
write me a haiku about invoices -> {"route": "human", "note": "no fallback configured; ask the user to confirm"}

The second line is the reason for the gate. The model picked the wrong tool entirely, but with low confidence, so the user would have seen a confirmation instead of the lounge lights coming on. The third request was correctly refused. Set FALLBACK_URL (for example http://gpu-box:8000/v1/chat/completions) and FALLBACK_MODEL to route refusals to a larger model.

Routing a Cactus response: calls with confidence at or above the threshold run locally; calls below it or suppressed calls go to user confirmation; no calls escalate to a larger model or a person.
Three outcomes, decided from fields the engine already returns.

7. Step 5: structured extraction that refuses invented values

cactus.extract() declares your record as the only tool and returns the fields. In strict mode (the default) it raises ExtractionValidationError if a value is not grounded in the text [2].

import cactus

INVOICE = {
    "name": "invoice",
    "description": "An invoice record",
    "parameters": {
        "type": "object",
        "properties": {
            "vendor": {"type": "string"},
            "total": {"type": "number"},
            "currency": {"type": "string"},
        },
        "required": ["vendor", "total"],
    },
}

for text in [
    "Invoice from Acme Ltd. Amount due: 1200 GBP.",
    "Invoice from Acme Ltd, total GBP 1,200.00, due 30 November.",
]:
    try:
        print("ok:", cactus.extract(text, INVOICE))
    except cactus.ExtractionValidationError as err:
        print("rejected:", err)
        print("  unchecked value would have been:", cactus.extract(text, INVOICE, strict=False))
ok: {'vendor': 'Acme Ltd. Amount', 'total': 1200.0}
rejected: extraction returned values not grounded in the input: total
  unchecked value would have been: {'vendor': 'Acme Ltd', 'total': 12200.0, 'currency': 'GBP'}

Two lessons from one small test. Strict mode caught a wrong total (12,200 instead of 1,200) that would otherwise have gone into your system. And the first result passed strict mode with a messy vendor name ("Acme Ltd. Amount"), because grounding checks that a value comes from the text, not that it is the right span. Keep your own validation for anything that matters.

8. Step 6: speech

Whistle needs 16 kHz mono WAV, or 16 kHz float samples. Make a test clip:

# macOS: synthesise a test clip and convert it to 16 kHz mono WAV
say -o lights.aiff "Turn the office lights down to forty percent."
afconvert -f WAVE -d LEI16@16000 -c 1 lights.aiff lights.wav

# Linux or Windows: convert any recording with ffmpeg
ffmpeg -i recording.m4a -ar 16000 -ac 1 lights.wav
import json
from typing import Literal

import cactus


@cactus.tool
def set_lights(room: Literal["kitchen", "lounge", "office"], brightness: int):
    """Set a room's light brightness.

    Args:
        room: which room
        brightness: brightness in percent, 0 to 100
    """
    return {"room": room, "brightness": brightness}


# 1. Plain transcription with word timestamps
t = cactus.transcribe("lights.wav", language="en", word_timestamps=True)
print(t["text"], "| ttft_ms:", t["ttft_ms"])
print([(w["word"] if "word" in w else w, w.get("start"), w.get("end")) for w in t["words"][:3]])

# 2. Speech straight to a tool call (no separate transcription step)
with cactus.Cactus(tools=[set_lights]) as agent:
    r = agent.complete(audio="lights.wav", language="en")
    print(r["audio_text"], "->", json.dumps(r["function_calls"]), r["confidence"])
Turn the office lights down to 40%. | ttft_ms: 298.1
[('Turn', 0.0, 0.24), ('the', 0.24, 0.4), ('office', 0.4, 0.64)]
Turn the office lights down to 40%. -> [{"name": "set_lights", "arguments": {"room": "office", "brightness": 40}}] 0.9515

Passing audio= to complete() goes from speech to a tool call in one step and returns the transcript as audio_text [2]. For a 5-second clip, transcription took 0.74 seconds once loaded. For live microphone input, cactus.stream() takes roughly one-second chunks and yields committed text as it goes; install cactus-compute[mic] for microphone capture and resampling [2].

Limits worth knowing: seven languages (English, German, French, Spanish, Italian, Dutch, Polish); silence returns empty text, but music and other noise can produce words, so put a voice activity detector or wake word in front of an always-on microphone [2]. Cactus reports Whistle beating Whisper base on several word error rate test sets and losing on others, at 16.9 MB against 145.3 MB [6]. We did not reproduce those benchmarks; our test clips were synthetic speech, which is easier than a real room.

9. Step 7: vision

import json

from PIL import Image, ImageDraw, ImageFont

from cactus import Mirage

# Make a test image so the example is self-contained
img = Image.new("RGB", (640, 400), "white")
draw = ImageDraw.Draw(img)
font = ImageFont.load_default(size=40)
draw.text((40, 40), "INVOICE 1042", fill="black", font=font)
draw.text((40, 120), "Total: GBP 1,200.00", fill="black", font=font)
draw.ellipse((420, 220, 560, 360), fill="red")
img.save("card.png")

vision = Mirage()

# Text lines with confidence (boxes are on a 0-1000 grid)
for line in vision.describe("card.png", sections=["text"])["text"]:
    print(f'{line["score"]:.2f}  {line["string"]}  {line["bbox"]}')

# Fill a small form with closed choices
form = {
    "document_type": {"enum": ["invoice", "receipt", "letter", "other"]},
    "has_red_circle": {"enum": ["yes", "no"]},
}
for field, answer in vision.extract("card.png", form)["form"].items():
    print(field, "->", answer["value"])
0.38  INVOICE  [66, 113, 321, 210]
0.70  INVOICE 1042  [60, 108, 464, 216]
0.50  1042  [321, 109, 472, 216]
0.37  Total: GBP  [55, 304, 369, 420]
0.67  Total: GBP 1,200.00  [52, 303, 620, 423]
0.51  1,200.00  [381, 310, 612, 418]
document_type -> invoice
has_red_circle -> None

Text reading worked, with overlapping candidates at different confidence levels; keep the highest-scoring line per region. Boxes come back on a 0 to 1,000 grid, not in pixels (our 640 by 400 image's red circle came back near x 650 to 880). The form answered one question and abstained on the other (None, with the probabilities in the full result). In an earlier run with a different font it answered the red-circle question wrongly. Mirage is useful for reading text and rough layout; for decisions, validate on your own images and gate on the returned probabilities.

10. Step 8: measure latency and memory

import os
import statistics
import time

import cactus

cactus.set_threads(int(os.environ.get("CACTUS_THREADS", "1")))


@cactus.tool
def set_lights(room: str, brightness: int):
    "Set a room's light brightness in percent."
    return {"room": room, "brightness": brightness}


QUERIES = ["dim the kitchen to 30 percent", "office lights to 80", "turn the lounge down to 10"]

t0 = time.perf_counter()
agent = cactus.Cactus(tools=[set_lights])
print(f"load: {time.perf_counter() - t0:.2f}s")

agent.complete(QUERIES[0])  # warm-up, not counted
wall, decode, ram = [], [], []
for i in range(30):
    agent.reset()
    t = time.perf_counter()
    r = agent.complete(QUERIES[i % len(QUERIES)])
    wall.append((time.perf_counter() - t) * 1000)
    decode.append(r["decode_tps"])
    ram.append(r["peak_ram_mb"])
agent.close()

wall.sort()
print(f"p50 {statistics.median(wall):.0f} ms  p95 {wall[int(0.95 * len(wall)) - 1]:.0f} ms")
print(f"decode median {statistics.median(decode):.0f} tok/s  peak RAM {max(ram):.0f} MB")

Run it with CACTUS_THREADS=1 python bench.py, then with 0 (all cores) and 4. Two of our runs:

CACTUS_THREADS=1: p50 105 ms  p95 112 ms   decode median 451 tok/s  peak RAM 136 MB
CACTUS_THREADS=0: p50 81 ms   p95 167 ms   decode median 534 tok/s  peak RAM 136 MB
Latency by thread count. Moderate load: all cores 81 ms, one thread 105 ms, four threads 129 ms. Saturated machine: one thread 110 ms, all cores 1,392 ms, four threads 1,475 ms with a p95 of 21.8 seconds.
On a quiet machine more threads helped a little; on a saturated one they made latency far worse. One thread stayed steady throughout.

Our Mac was shared with other heavy jobs, which proved useful. When other work saturated the cores, the default setting pushed the median to 1.4 seconds and four threads gave a 22-second p95, while one thread stayed at about 110 ms. A phone that is also rendering UI can hit the same contention. Start with one or two threads and measure on the slowest device you support. peak_ram_mb in each response is the engine's own figure; the whole Python process peaked at about 176 to 197 MB in our runs.

11. Step 9: package it to run offline

Nothing should download on a customer's device or in production. Fetch the engine and weights at build time and point the package at them [3].

A container that runs with networking off

FROM python:3.12-slim

ENV CACTUS_TELEMETRY=0 \
    XDG_CACHE_HOME=/opt/cache \
    PIP_NO_CACHE_DIR=1

RUN pip install "cactus-compute==4.2.2"

# Fetch the engine and the weights at build time; nothing downloads at run time.
RUN cactus fetch \
 && cactus download needle3 --out /opt/models \
 && chmod -R a+rX /opt/cache /opt/models

ENV HF_HUB_OFFLINE=1 \
    CACTUS_NEEDLE_WEIGHTS=/opt/models/needle3.cact

RUN useradd --create-home app
USER app
WORKDIR /app
COPY app.py .
CMD ["python", "app.py"]
docker build -t cactus-edge-demo .
docker run --rm --network none cactus-edge-demo

We built this on Apple silicon (so a Linux arm64 image of about 134 MB) with gated.py as app.py, and it printed the same three routes as step 4 with networking disabled. One trap we hit: cactus download needle3 alone writes into the current directory, so without --out and CACTUS_NEEDLE_WEIGHTS the non-root user tried to download the weights again at run time and failed on a read-only cache.

Build time with internet: pip install a pinned version, cactus fetch, cactus download needle3. Run time with no network: HF_HUB_OFFLINE=1, CACTUS_TELEMETRY=0 and CACTUS_NEEDLE_WEIGHTS pointing at the bundled file.
Fetch at build time, run with the network off.

An air-gapped Linux device

# On a connected machine with the same OS, CPU architecture and Python as the device
pip download "cactus-compute==4.2.2" -d wheels
cactus fetch --platform-tag manylinux2014_aarch64 --out ./engine   # Linux Arm64 engine
cactus download needle3 --out ./models

# Copy wheels/, engine/ and models/ to the device, then on the device:
pip install --no-index --find-links wheels cactus-compute
export HF_HUB_OFFLINE=1 CACTUS_TELEMETRY=0
export CACTUS_LIB_PATH=$PWD/engine/libcactus.so
export CACTUS_NEEDLE_WEIGHTS=$PWD/models/needle3.cact

We checked the equivalent on macOS with an empty cache directory, HF_HUB_OFFLINE=1 and the two environment variables set: the tool call ran without touching the network. Use CACTUS_WHISTLE_WEIGHTS and CACTUS_MIRAGE_WEIGHTS for the other models [3].

Mobile and browser apps

For apps outside Python, cactus download <platform> --model needle3 fetches a native build: iOS, tvOS and watchOS get a static library and a C header (cactus.h); Android and Linux also get a command-line runner; the browser target is WebAssembly [5]. We did not build a mobile app for this note; follow Cactus's deployment guide and treat it as a C library you wrap in Swift or Kotlin.

12. Test and evaluate before you ship

The only accuracy figures that matter are the ones on your requests. Write a test file of real queries with the expected call, including ones that should be refused, then score it at several thresholds:

{"query": "dim the kitchen to 20 percent", "expect": {"name": "set_lights", "arguments": {"room": "kitchen", "brightness": 20}}}
{"query": "the wifi in the office keeps dropping", "expect": {"name": "create_ticket"}}
{"query": "what's the capital of France?", "expect": null}
import json
import sys

import cactus

from gated import TOOLS  # reuse the tool definitions


def matches(call, expect):
    if expect is None:
        return call is None
    if call is None or call["name"] != expect["name"]:
        return False
    wanted = expect.get("arguments", {})
    return all(call["arguments"].get(k) == v for k, v in wanted.items())


cases = [json.loads(line) for line in open(sys.argv[1])]
rows = []
with cactus.Cactus(tools=list(TOOLS.values()), system="locale: en-GB") as agent:
    for case in cases:
        agent.reset()
        r = agent.complete(case["query"])
        call = r["function_calls"][0] if r["function_calls"] else None
        rows.append((r["confidence"] or 0.0, call is not None, matches(call, case["expect"]), case))

for threshold in (0.0, 0.5, 0.7, 0.9):
    acted = [row for row in rows if row[1] and row[0] >= threshold]
    wrong = [row for row in acted if not row[2]]
    missed = sum(1 for row in rows if row[3]["expect"] is not None and row not in acted)
    print(f"threshold {threshold:.1f}: acted on {len(acted)}/{len(rows)}, "
          f"wrong actions {len(wrong)}, not acted (sent to confirm/fallback) {missed}")
for conf, has_call, ok, case in rows:
    print(f"{conf:.2f} {'ok ' if ok else 'BAD'} {case['query']}")

On our seven-case file (save it as cases.jsonl beside gated.py and run python evaluate.py cases.jsonl):

threshold 0.0: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.5: acted on 4/7, wrong actions 0, not acted (sent to confirm/fallback) 1
threshold 0.7: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
threshold 0.9: acted on 3/7, wrong actions 0, not acted (sent to confirm/fallback) 2
0.95 ok  dim the kitchen to 20 percent
0.97 ok  office lights to 75
0.96 ok  make the lounge brighter, 90 percent
0.22 BAD the wifi in the office keeps dropping
0.62 ok  log an urgent ticket: server room is overheating
0.96 ok  what's the capital of France?
0.90 ok  write me a poem

Seven cases prove nothing statistically; 100 to 200 real ones per tool surface is a sensible start. The shape is what matters: raising the threshold trades actions for confirmations. Pick the point where wrong actions are rare enough for the task, and rerun the file in CI whenever you change the package version, weights or tool definitions. Cactus ships six ready-made tool surfaces with acceptance suites in cactus.environments, and notes that the base model does not pass five of them at the default gate [2], a useful reminder that a small model's fit depends on the tools.

If the base model is not accurate enough on your tools, Cactus supports LoRA fine-tuning (a cheap way to adapt a model by training small extra weight matrices) locally with cactus-compute[train], or on its paid hosted platform, producing a new .cact you load with weights= [2]. Note that locally fine-tuned weights do not include the calibration head, so confidence comes back as None [2]; you would need another gate.

13. Running it in production

  • Pin everything. Package version, and the weights file you tested with. Weekly releases are normal for this project.
  • Threads. Set cactus.set_threads() explicitly, before loading models; start at one or two on shared or mobile CPUs.
  • Concurrency. Text, speech and vision share one runtime and calls are serialised across threads [3]. For a server handling many requests, run several worker processes rather than threads.
  • Security. The model chooses which of your functions to call. Only expose functions that are safe to run with any schema-valid arguments, and re-check arguments and permissions in the function itself.
  • Monitoring. Log route (local, confirm, fallback), confidence, latency, peak_ram_mb and whether the user corrected the action. Avoid logging transcripts or extracted personal data unless you need them and have a basis to keep them.
  • Privacy. Disable telemetry where your policy requires it, and remember that the fallback path sends data off the device: apply the same data rules there.

14. Pitfalls and debugging

  • Empty function_calls everywhere. The request does not map to a tool, or the tool description is unclear. Check the reasoning field and rewrite the docstring.
  • confidence is None. You loaded locally fine-tuned weights without the calibration head [2].
  • ExtractionValidationError. A value was not found in the text. Do not switch to strict=False to make it go away; inspect it.
  • PermissionError on start in a container. The package is trying to download into a cache it cannot write. Bundle the files and set CACTUS_NEEDLE_WEIGHTS and HF_HUB_OFFLINE=1.
  • Latency spikes. CPU contention with multi-threading. Try set_threads(1).
  • Wrong sample rate. Whistle expects 16 kHz mono; convert, or install the [mic] extra for resampling [1].

15. How it compares with the alternatives

OptionWhat you runTypical sizeBest forTrade-offs
Cactus (cactus-compute)Cactus's own Needle, Whistle and Mirage models on one engine17 to 35 MB per modelNarrow tool calls, extraction, short speech, OCR on any CPU, offlineNo free text; small tool sets; young project; few independent benchmarks
llama.cpp (and Ollama on top)Open LLMs and some vision models in GGUF format [11]Hundreds of MB to tens of GBGeneral chat, drafting and tool use with 1B to 70B+ models on laptops and serversFar more memory; tool-calling reliability varies by model
MLC LLMOpen LLMs compiled for phones, browsers and GPUs [14]Hundreds of MB and upChat-class models inside mobile and web appsCompilation step per model and target
ONNX RuntimeYour own model exported to ONNX [12]Whatever you exportShipping a custom-trained model across platforms and acceleratorsYou own training, export and pre/post-processing
ExecuTorchYour PyTorch model, exported for edge devices; 1.0 since October 2025 [13]Whatever you exportPyTorch teams targeting phones and embedded NPUsMore engineering; you bring the model
NVIDIA TensorRT Edge-LLMLLMs, vision-language and speech models on Jetson, DRIVE and DGX Spark [10]GB-class modelsRobots, vehicles and edge boxes with NVIDIA GPUs; NVIDIA's own Nemotron modelsNVIDIA hardware only; engines built per device

NVIDIA's own on-device direction is larger models on NVIDIA edge hardware: its Nemotron 3 Nano Omni, for example, is a 30B-parameter mixture-of-experts model with about 3B active per token, aimed at Jetson and DGX Spark among other targets [15]. That is a different class from a 35 MB CPU model. A sensible architecture often uses both: a tiny model on the user's device for the frequent, narrow requests, and a larger model on a GPU server you control for the fallback.

16. FAQ

Is this an NVIDIA product? No. Cactus Compute builds and publishes it under Apache 2.0, including the prebuilt runtime binaries [1]. The nearest NVIDIA link is that Cactus also ships a Mac runner, cactus-mlx, for compressed checkpoints of NVIDIA's Nemotron models [1].

Can Needle answer questions or summarise text? No. It returns tool calls or extracted fields only. Use a larger model for prose.

Which languages does speech support? English, German, French, Spanish, Italian, Dutch and Polish; fine-tuning can add others [2].

Can I use it commercially? The code and runtime binaries are Apache 2.0 [1], and the Mirage model card lists Apache 2.0 [7]. Check the licence of each model you ship on its Hugging Face card.

If you are deciding which AI work should run on devices, on your own GPU servers or in the cloud, and need the private side built and run, see how we approach private AI.

References

  1. cactus-compute/cactus: README, GitHub, Cactus Compute.
  2. Cactus API reference (llms.txt), Cactus Compute.
  3. Needle Python API and CLI documentation, Cactus Compute.
  4. Needle supported devices, Cactus Compute.
  5. Cactus runtime binaries and platforms, Hugging Face.
  6. Whistle announcement, Cactus Compute, 2 October 2026.
  7. Mirage model card, Hugging Face.
  8. cactus-compute on PyPI (release history).
  9. Cactus Compute company page, Y Combinator.
  10. TensorRT Edge-LLM overview, NVIDIA.
  11. llama.cpp, GitHub.
  12. ONNX Runtime, Microsoft.
  13. Introducing ExecuTorch 1.0, PyTorch.
  14. MLC LLM, GitHub.
  15. NVIDIA Nemotron 3 Nano Omni, NVIDIA blog, 28 April 2026.

All code in this note was run by Workstation on 16 November 2026 with cactus-compute 4.2.2 on an Apple M4 Max and in a Linux arm64 container; outputs are copied from those runs. Performance figures are indicative, from a shared machine. Other figures are vendor claims. This note is independent and not endorsed by Cactus Compute or NVIDIA. NVIDIA, Jetson, DRIVE, DGX, Nemotron and TensorRT are trademarks of NVIDIA Corporation. Other names are the property of their respective owners.

แชร์บทความนี้

เพิ่มเติมใน Technology

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Testing ML and LLM Systems You Can't Prove Correct: A Python Evidence Kit

Evidence instead of proofs, in about 400 lines of Python with real output: slice metrics with bootstrap intervals and a pass/fail contract, Hypothesis property and metamorphic tests, calibration and conformal prediction, Pydantic contracts and regression evals for LLM output, drift checks, a CI gate, and exact verification of a small neural network

อ่านต่อ
TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

TorchCurves in Practice: Learnable B-spline and Legendre Curves in PyTorch

Install torchcurves, fit and size a curve, choose B-spline or Legendre and the input map, build a monotone calibration curve and an additive model on tabular data, compare with SciPy, scikit-learn, a KAN and an MLP, export coefficients to JSON, test and run it in production, with real outputs

อ่านต่อ
AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

AI Usage Policy as Code: Build an LLM Gateway with PII Redaction, Tool Allow-Lists and Telemetry

Close the AI guidance gap in code: a versioned policy file, an OpenAI-compatible FastAPI gateway, Presidio redaction, per-team tool and MCP allow-lists, red-team tests, an audit log, Prometheus metrics and a usage report

อ่านต่อ