NVIDIA Cybersecurity AI with OpenTelemetry and SIEM: BlueField, DOCA Argus, OpenShell and Guardrails into Elastic Security, Dynatrace, Prometheus and Grafana

A practitioner's guide to securing AI data centres and AI agents with NVIDIA's cybersecurity portfolio, and to watching all of it through one OpenTelemetry pipeline that feeds the tools security teams already run: Elastic Security as the SIEM, Dynatrace, Prometheus with Alertmanager, and Grafana. It covers the threats specific to GPU estates and agents, what each NVIDIA product does layer by layer (BlueField and DOCA, DOCA Argus, DOCA Vault, Sentry, confidential computing, OpenShell, Nemotron Safety, NeMo Guardrails, garak and Morpheus), the telemetry each one actually emits, a validated Collector configuration, fourteen Elastic detection rules that were loaded into a live Kibana and fired against test events, Prometheus alert rules with unit tests, response playbooks, compliance mapping, retention and cost, high availability, and how Workstation delivers it.

This article extends two earlier technotes rather than replacing them. The UFM, DCGM and OpenTelemetry reference design sets out the Collector agent on every node and the gateway pair we use for operational telemetry, and the Minerva AIOps runbook covers incident correlation and change-controlled remediation. Security telemetry follows the same agent-and-gateway pattern, with one change: it gets its own gateway pair, owned by the security team. Our Big Prometheus and MoSCoW standard still applies: OpenTelemetry is the telemetry contract, PromQL is the metrics language, and each kind of log has one primary store.

1. Executive summary

AI data centres are attractive targets for reasons ordinary server rooms are not. The hardware is expensive and easy to steal time from. The model weights and training data are often the most valuable intellectual property a company owns. And AI agents now act on their own: they read untrusted web pages and documents, call tools, use credentials and change systems. A single hidden instruction in a document can turn a helpful agent into an insider.

NVIDIA's answer is defence in depth across the stack. In plain terms:

• The DPU watches the server from outside it. BlueField runs security services in its own isolated domain, so malware on the host cannot switch them off. DOCA Argus inspects host memory for runtime threats; Sentry monitors agents and can quarantine them in milliseconds; DOCA Vault protects sensitive data.
• The GPU protects data while it is being used. Confidential computing keeps data and model weights encrypted and isolated inside the GPU, and attestation proves it before anything sensitive is released.
• Agents run in a sandbox with a written policy. OpenShell decides which files, network destinations, processes and credentials an agent may use, enforces that in the kernel and proxy, and logs every decision in a standard security schema.
• Models are checked on the way in and out. Nemotron Safety models and NeMo Guardrails filter unsafe prompts and responses; garak attacks your agents before attackers do.

What NVIDIA's products do not do on their own is give a security operations team one place to see all of this, correlate it with identity, Kubernetes, the web edge and secrets, and respond. That is the part this article is about, and it is where Workstation's observability and SIEM practice comes in. The outcome for the business is:

• every security control produces evidence that lands in your SIEM within seconds, in one schema;
• detections for the attacks that matter in AI estates (agent hijacking, model theft, GPU misuse, privilege abuse), each with a tested response;
• containment that is automatic only when it is safe and reversible, with human approval otherwise;
• evidence for auditors under UK GDPR, ISO 27001 and NIS-style regimes, at a predictable cost.

Workstation is an NVIDIA partner. We supply the hardware, and we provide the engineering, software and people to design, build and run this with you, from a first assessment to ongoing monitoring support (section 17).

2. The threat landscape for AI data centres and agents

Most threats to an AI estate are familiar ones with new targets. The table maps each to where it shows up, which control prevents or limits it, and which signal tells you it happened. Section 11 turns each signal into a detection rule.

3. NVIDIA's cybersecurity portfolio, layer by layer

NVIDIA groups these products on its cybersecurity AI solutions page. We find it clearer to arrange them by the layer they protect, because that tells you where each one's telemetry comes from and what it can see that the others cannot.

3.1 Silicon and DPU: BlueField, DOCA, DOCA Argus, DOCA Vault and Sentry

BlueField is NVIDIA's data processing unit: a network adapter with its own Arm cores, memory and operating system, sitting between the server and the network. DOCA is the software framework for programming it. The security point is separation. Services on the DPU run in a domain the host operating system does not control, so an attacker with root on the host still cannot read or stop them.

DOCA Argus uses that position to watch the host. According to NVIDIA's DOCA Argus service guide, it runs as a container on BlueField-3 or later (in DPU mode), reads selected kernel structures from host memory over DMA, and rebuilds a picture of processes, binaries with their hashes, command lines, files, network connections, containers and VMs, and GPU use per process, with no agent installed on the host. From that it raises alerts for runtime threats such as reverse shells and unexpected executions. What matters for integration:

• Messages have a message_type of EVENT, ALERT or SYSTEM_ACTIVITY and a severity from INFO to CRITICAL, with payload sections such as workload_information and activity_data.
• Telemetry export is off by default. It is configured with telemetry_address, telemetry_format (JSON or syslog) and telemetry_tag, and NVIDIA's examples send it over TCP to Fluent Bit or Vector.
• NVIDIA states that Argus integrates with leading enterprise XDR platforms. If you run one, it can stay the primary consumer; our pipeline then receives a copy for correlation with everything else.

Our gateway receives Argus JSON directly on a TCP listener (section 6). If your Argus release does not separate messages with newlines, put Fluent Bit or Vector in between as NVIDIA's guide shows and forward to the Collector over OTLP; the rest of the design is unchanged. NVIDIA's guide shows the message fields at the top level in its example but describes a message_header root in its schema table, so check a real message from your release before relying on the field paths in our rule.

DOCA Vault is a security microservice on BlueField that guards sensitive data with in-silicon controls and zero-trust access. We have not tested it, and we do not describe its interfaces here. In our designs it sits alongside, not instead of, the key management that releases secrets to workloads (for us, wslvault).

NVIDIA Sentry on BlueField is the newest piece. NVIDIA describes it as part of its Open Agent Safety Platform: running in BlueField-4's hardware-isolated domain, independent of the host and of the agent's own software, it monitors agent behaviour for policy deviations and suspicious activity, provides trusted telemetry, and can quarantine an agent in milliseconds using the DPU's infrastructure controls. Because it is out of band, it still works if the host or the agent runtime has been compromised, which is exactly when you need it. We treat Sentry's alerts and quarantine actions as first-class SIEM events; the exact export format should be taken from NVIDIA's Sentry documentation for the release you deploy.

DOCA Telemetry Service exports DPU counters in Prometheus format. It is already part of the UFM reference design and needs no change here, except that port and drop counters on DPUs in front of agent hosts become useful context in security investigations.

3.2 Platform: confidential computing and attestation

Confidential computing protects data and models in use. The GPU runs the workload inside a hardware trusted execution environment, so that neither the host operating system, the hypervisor nor the people who run the infrastructure can read the memory, and the GPU can prove its state through attestation. That makes it the control for two threats nothing else in the table addresses well: an untrusted hosting provider, and sharing data or models with a party you do not fully trust.

The operational rule is simple: no passing attestation, no keys. A launch wrapper attests the GPU (and the confidential VM) before the workload starts and only then asks the key service for the dataset or model decryption key. NVIDIA publishes the attestation tooling in nvTrust; its README marks the Python attestation SDK and local GPU verifier as deprecated in favour of a C++ SDK and CLI, so plan new integrations on those. We log each attestation result as a structured event (pass or fail, which GPU, which workload, which verifier) and alert on every failure.

3.3 Runtime: OpenShell

OpenShell is NVIDIA's open-source (Apache 2.0) runtime for autonomous agents. Each agent runs in a sandbox; a policy declares the files it may read or write, the network destinations and HTTP methods it may use, the processes it may run and the credentials it may use, and OpenShell enforces that in the kernel and in a policy-checking proxy. Real credentials stay outside the sandbox: OpenShell injects them into outgoing requests, and only when the destination is an approved endpoint. Before a policy change is applied, a formal verification step flags new access that needs human review. Our earlier OpenShell article walks through running agents under it.

For a SIEM, OpenShell is the richest source in the stack, because it records every security decision in the Open Cybersecurity Schema Framework (OCSF) version 1.8.0. From its observability documentation:

• Sandbox supervisors and the gateway emit OCSF events for network activity (class 4001), HTTP activity (4002), SSH (4007), process activity (1007), detection findings (2004), authentication (3002), configuration changes (5019), application lifecycle (6002) and API activity (6003).
• Denied network and HTTP events carry the reason in status_detail, for example no matching policy or resolves to always-blocked address (a link-local or loopback destination such as a cloud metadata endpoint).
• When a sandbox credential calls a method or another sandbox it should never touch, the gateway raises a high-severity detection finding with finding_info.uid sandbox-authorization-boundary: the credential may be compromised.
• Setting enabled = true under [openshell.gateway.otlp.ocsf] in gateway.toml exports OCSF events as OTLP log records under the instrumentation scope openshell.ocsf. The body is the OCSF event as JSON; attributes include ocsf.class_uid, ocsf.severity_id, ocsf.metadata.uid and openshell.sandbox.id; supervisors send their own events to the same endpoint.
• Delivery is at least once, so deduplicate on ocsf.metadata.uid. Records the gateway drops are counted in openshell_ocsf_otlp_dropped_total, and the next batch carries a warning record named openshell.ocsf.gap.
• The gateway serves Prometheus metrics on a separate listener, unauthenticated, so keep it internal; interceptor failures that let a request through are counted in openshell_gateway_interceptor_fail_open_total.

The shorthand log line for a denied connection looks like this (from NVIDIA's documentation):

OCSF NET:OPEN [MED] DENIED /usr/bin/curl(64) -> httpbin.org:443 [policy:- engine:opa] [reason:no matching policy]

NVIDIA's OpenShell Reference Architecture builds a full stack around the runtime: Vera CPUs run the agent harnesses and their tools, Vera Rubin systems serve inference, and Sentry on BlueField-4 adds monitoring and enforcement that does not depend on the host. NVIDIA is explicit that OpenShell also runs on other CPU systems and that Vera and BlueField are optional; they add efficiency and the out-of-band layer, not basic function. NemoClaw, NVIDIA's collection of open blueprints for specialised agents built on Nemotron models, runs its agents inside OpenShell, so everything in this article applies to NemoClaw agents unchanged; we cover NemoClaw itself in a separate article.

3.4 Models and guardrails: Nemotron Safety, NeMo Guardrails and garak

Nemotron is NVIDIA's family of open models; NVIDIA positions some for planning and reasoning, some for fast, high-volume tool calls, and a set of Nemotron Safety models that classify unsafe content in text and images so it can be filtered on the way into or out of an agent pipeline. Specialised with NeMo, Nemotron models can also run the defenders' own agents for alert triage, continuous threat hunting and red-team simulation.

NeMo Guardrails (latest release 0.24.1 at the time of writing) puts programmable rails between an application and its model: input rails (for example jailbreak checks and masking sensitive data), dialog rails, retrieval rails on RAG chunks, execution rails on tool inputs and outputs, and output rails. A safety classifier such as a Nemotron Safety model can sit behind a rail. For monitoring, its config.yml has a tracing section with an OpenTelemetry adapter; the library uses only the OpenTelemetry API and your application configures the SDK and exporter. Prompt and response content is not captured unless you set enable_content_capture: true; leave it off in production, because prompts often contain personal data.

Spans tell you which rails ran and how long they took, which is ideal for Dynatrace. For the SIEM we also want one event per block, with who, which rail and what happened, but no content. We emit that from a thin wrapper around the rails call, using our own field convention (event.action: input_rail_blocked, rule.name set to the rail flow, user.id). That convention is Workstation's, not part of NeMo Guardrails.

garak is NVIDIA's LLM vulnerability scanner: it runs probes (prompt injection, jailbreaks such as the DAN family, data leakage, encoding tricks, training-data replay and more) against a model or agent endpoint, and scores the responses with detectors. Each run writes a JSONL report; its evaluation records have entry_type: "eval" with probe, detector, passed, fails and total_evaluated. We run garak in CI on every change to a model, prompt or guardrail configuration and ship those records to the SIEM, so a regression is a detection like any other.

# CI step (example): scan the staging endpoint with a fixed probe set
python -m garak --config ci/garak-staging.yaml --target_type rest \
  --spec probes.promptinject,probes.dan,probes.encoding,probes.leakreplay \
  --report_prefix ci-${CI_PIPELINE_ID}

Here ci/garak-staging.yaml is a garak configuration file that gives the REST generator your endpoint's URI and request template (see garak's documentation; it also has generators for OpenAI-compatible endpoints, Hugging Face, Bedrock and others). Current garak releases select probes with --spec; the older --probes flag still works but is deprecated. Copy the resulting *.report.jsonl into the directory the CI runner's Collector watches.

3.5 Pipelines: Morpheus

Morpheus is NVIDIA's GPU-accelerated framework for building cybersecurity inference pipelines over large data streams. Its repository ships example pipelines and pre-trained models for digital fingerprinting of user and machine behaviour, phishing detection, sensitive information detection, log parsing, ransomware detection from AppShield data, fraud detection with graph neural networks, data loss prevention, and anomalous behaviour profiling, including one example that detects cryptocurrency mining from GPU telemetry. Check its status before planning around it: at the time of writing the latest tagged release is 25.06 (July 2025). Where a customer already has Morpheus pipelines, their outputs are just another scored source: we ship the scores and flagged records into the same gateway as events.

4. Workstation's security observability reference architecture

Five principles drive the design:

1. One pipeline. Every security signal enters through an OpenTelemetry Collector. Backends never pull from sources directly, so adding a backend or replacing one is a gateway change, not a fleet change.
2. Normalise once, at the gateway. Fields are mapped to ECS names (and OTel semantic conventions where they exist) before data forks, so a rule in Elastic and a dashboard in Grafana are talking about the same destination.domain.
3. Redact before anything leaves. Authorisation headers, cookies, API tokens and prompt or completion text are removed at the gateway. No backend ever stores them, so no backend needs to be trusted with them.
4. Each backend does what it is best at. Elastic Security holds every security event and runs correlation rules and cases. Dynatrace gets the higher-severity events, metrics and traces so its AI can tie security events to service health. Prometheus gets low-cardinality counters and the GPU and network metrics it already holds, for fast, cheap alerting. Grafana is the shared screen.
5. The pipeline watches itself. Dropped records, failing exports and lost gateways are security alerts, not operations tickets (section 15).

Tiers

• Node agents are the same agents as in the UFM design: one Collector per GPU, storage, login and control-plane node. We add three receivers (auditd, the Kubernetes API server audit log, and garak reports on CI runners) and a second exporter that sends security logs to the security gateways rather than the operations gateways.
• A security gateway pair (or more), behind a virtual IP or DNS name, run by the security team. It receives OTLP from agents, from OpenShell gateways and supervisors and from guarded applications; Argus JSON over TCP; and syslog over TLS from appliances. It scrapes the OpenShell gateway and wslproxy metrics endpoints. Backend credentials exist only here.
• The operations gateways are unchanged. DCGM, node and fabric metrics still flow to Mimir or Prometheus as in the UFM design; the cryptojacking and exfiltration alerts in section 9 run there, because that is where the data already is.

Why separate security gateways? Separation of duties (the platform team cannot quietly filter out security events), different retention and access rules, and a smaller blast radius: a misbehaving dashboard query or an operations change cannot starve the security pipeline.

Sources

The wafsec field list comes from our WAF Engine v2 article. For wslvault and OpsAPI we configure the products to log ECS-named JSON (event.action, event.outcome, user.id, source.ip), which keeps the gateway mapping trivial; the action names used in this article's rules (secret.read, login, agent.approval.granted) are our convention, so adjust them to what your deployment emits.

5. Normalisation: OCSF, ECS and OpenTelemetry semantic conventions

Three schemas meet in this pipeline. OpenShell speaks OCSF; Elastic Security's rules, timelines and prebuilt content expect ECS; and OpenTelemetry has its own semantic conventions for resources and attributes. We keep the original record intact for forensics and add ECS fields beside it. The gateway does the mapping that every backend needs (so Dynatrace sees the same names as Elastic), and an Elasticsearch ingest pipeline does the Elastic-specific enrichment.

OpenShell's documentation notes that sandbox identity moved from a top-level container object to device.container in OCSF 1.8.0 records, and warns SIEM parsers to follow. Our mapping reads the new location only, so pin the OpenShell version in your change records and re-test the mapping on upgrade.

6. Collector configuration

All configuration below uses component names from otelcol-contrib 0.165.0, which introduced snake_case names (file_log, tcp_log, otlp_grpc, otlp_http, prometheus_remote_write, cumulative_to_delta, delta_to_cumulative, resource_detection). The older names in our UFM and Minerva articles still work as deprecated aliases and log a warning at start-up. Secrets are environment placeholders.

Node agent: security receivers

These pipelines are added to the existing node agent. On a node that is not a control plane or a CI runner, the receivers for files that do not exist simply idle.

# /etc/otelcol-contrib/security-agent.yaml  (added to the node agent)
extensions:
  file_storage:
    directory: /var/lib/otelcol/queue
    create_directory: true

receivers:
  # Linux audit (auditd): one record per line, key=value pairs
  file_log/auditd:
    include: [/var/log/audit/audit.log]
    start_at: end
    storage: file_storage
    operators:
      - type: regex_parser
        regex: '^type=(?P<record_type>\S+) msg=audit\((?P<epoch>[\d.]+):(?P<sequence>\d+)\): ?(?P<kv>.*)$'
        parse_to: attributes["auditd.log"]
      - type: key_value_parser
        parse_from: attributes["auditd.log"]["kv"]
        parse_to: attributes["auditd.data"]
        on_error: send
      - type: remove
        field: attributes["auditd.log"]["kv"]
      - type: add
        field: attributes["data_stream.dataset"]
        value: auditd.log

  # Kubernetes API server audit log (control-plane nodes only)
  file_log/k8s-audit:
    include: [/var/log/kubernetes/audit/audit.log]
    start_at: end
    storage: file_storage
    operators:
      - type: json_parser
        parse_to: attributes["kubernetes.audit"]
      - type: add
        field: attributes["data_stream.dataset"]
        value: kubernetes.audit_logs

  # garak eval records from CI runners (one report per run)
  file_log/garak:
    include: [/var/lib/ci/garak/*.report.jsonl]
    start_at: beginning
    storage: file_storage
    operators:
      - type: json_parser
        parse_to: attributes["garak"]
      - type: filter
        expr: 'attributes.garak.entry_type != "eval"'
      - type: add
        field: attributes["data_stream.dataset"]
        value: garak.eval
      - type: add
        field: attributes["ci.pipeline"]
        value: EXPR(env("CI_PIPELINE_NAME"))

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resource_detection:
    detectors: [env, system]
  batch:
    send_batch_size: 2048
    timeout: 5s

exporters:
  otlp_grpc/security-gateway:
    endpoint: otel-security.sec.internal:4317
    tls:
      ca_file: /etc/otelcol-contrib/ca.pem
    sending_queue:
      enabled: true
      storage: file_storage

service:
  extensions: [file_storage]
  pipelines:
    logs/security:
      receivers: [file_log/auditd, file_log/k8s-audit, file_log/garak]
      processors: [memory_limiter, resource_detection, batch]
      exporters: [otlp_grpc/security-gateway]

Notes:

• The auditd parser splits the record header from its key-value pairs. It does not decode hex-encoded fields such as proctitle; if you need them, decode in the ingest pipeline. Enable the audit rules you want (for example execve on GPU nodes) in your auditd configuration; the Collector only ships what auditd writes.
• The Kubernetes audit receiver only sees what your API server audit policy records. To catch privileged pods you need at least RequestResponse level, or Request level, for pods create, update and patch, so the request body is in the log.
• The garak receiver keeps only evaluation records and labels them with the CI pipeline, so a regression points at the change that caused it.

Security gateway

# /etc/otelcol-contrib/security.yaml  (security gateway pair)
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage:
    directory: /var/lib/otelcol/queue
    create_directory: true
  basicauth/es:
    client_auth:
      username: ${env:ES_USER}
      password: ${env:ES_PASSWORD}

receivers:
  # Agents, OpenShell gateway and supervisors (OCSF over OTLP),
  # NeMo Guardrails spans and CI jobs all arrive here
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol-contrib/tls/gateway.crt
          key_file: /etc/otelcol-contrib/tls/gateway.key
      http:
        endpoint: 0.0.0.0:4318
        tls:
          cert_file: /etc/otelcol-contrib/tls/gateway.crt
          key_file: /etc/otelcol-contrib/tls/gateway.key

  # DOCA Argus telemetry (JSON lines over TCP from each BlueField)
  tcp_log/argus:
    listen_address: 0.0.0.0:5170
    operators:
      - type: json_parser
        parse_to: attributes["argus"]
      - type: add
        field: attributes["data_stream.dataset"]
        value: doca.argus

  # wslproxy WAF decisions (wafsec), wslvault and OpsAPI SSO audit, appliance syslog
  syslog/security:
    tcp:
      listen_address: 0.0.0.0:6514
      tls:
        cert_file: /etc/otelcol-contrib/tls/gateway.crt
        key_file: /etc/otelcol-contrib/tls/gateway.key
    protocol: rfc5424

  # Security-relevant metrics that live off-node: OpenShell gateway, wslproxy
  prometheus/security:
    config:
      scrape_configs:
        - job_name: openshell-gateway
          scrape_interval: 30s
          kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: [openshell]
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_container_port_name]
              regex: metrics
              action: keep
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
        - job_name: wslproxy
          scrape_interval: 30s
          file_sd_configs:
            - files: ["/etc/otelcol-contrib/targets/wslproxy-*.yaml"]
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "waf_(blocked|monitored|inspections|latency|errors).*"
              action: keep

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20

  resource/security:
    attributes:
      - key: site
        value: ${env:WS_SITE}
        action: upsert
      - key: cluster
        value: ${env:WS_CLUSTER}
        action: upsert

  # OpenShell OCSF records: body is the OCSF event as a JSON string.
  # Lift the fields every backend needs into attributes with ECS names,
  # tag the dataset and ingest pipeline, and use the OCSF event id as the
  # Elasticsearch document id so at-least-once retries do not duplicate.
  transform/ocsf:
    error_mode: ignore
    log_statements:
      - conditions:
          - scope.name == "openshell.ocsf"
        statements:
          - set(log.cache["ocsf"], ParseJSON(log.body)) where IsString(log.body)
          - set(log.attributes["event.action"], log.cache["ocsf"]["action"])
          - set(log.attributes["event.outcome"], "failure") where log.cache["ocsf"]["status_id"] == 2
          - set(log.attributes["event.outcome"], "success") where log.cache["ocsf"]["status_id"] == 1
          - set(log.attributes["destination.domain"], log.cache["ocsf"]["dst_endpoint"]["domain"])
          - set(log.attributes["destination.port"], Int(log.cache["ocsf"]["dst_endpoint"]["port"])) where log.cache["ocsf"]["dst_endpoint"]["port"] != nil
          - set(log.attributes["process.executable"], log.cache["ocsf"]["actor"]["process"]["name"])
          - set(log.attributes["container.name"], log.cache["ocsf"]["device"]["container"]["name"])
          - set(log.attributes["rule.name"], log.cache["ocsf"]["firewall_rule"]["name"])
          - set(log.attributes["ocsf.status_detail"], log.cache["ocsf"]["status_detail"])
          - set(log.attributes["data_stream.dataset"], "openshell.ocsf")
          - set(log.attributes["elasticsearch.ingest_pipeline"], "ws-ocsf-to-ecs")
          - set(log.attributes["elasticsearch.document_id"], log.attributes["ocsf.metadata.uid"])

  # Syslog from our own appliances: wslproxy's wafsec JSON is mapped field by
  # field; wslvault, OpsAPI SSO and the attestation wrapper already log
  # ECS-named JSON, so it is merged as it is and only the dataset is tagged.
  transform/appliances:
    error_mode: ignore
    log_statements:
      - conditions:
          - log.attributes["appname"] == "wafsec"
        statements:
          - set(log.attributes["wafsec"], ParseJSON(log.attributes["message"]))
          - set(log.attributes["data_stream.dataset"], "wslproxy.waf")
          - set(log.attributes["event.action"], log.attributes["wafsec"]["action"])
          - set(log.attributes["source.ip"], log.attributes["wafsec"]["client_ip"])
          - set(log.attributes["rule.id"], log.attributes["wafsec"]["signature_id"])
          - set(log.attributes["url.path"], log.attributes["wafsec"]["uri"])
          - delete_key(log.attributes, "message")
      - conditions:
          - log.attributes["appname"] == "wslvault" or log.attributes["appname"] == "opsapi-sso" or log.attributes["appname"] == "cc-attest"
        statements:
          - merge_maps(log.attributes, ParseJSON(log.attributes["message"]), "upsert")
          - set(log.attributes["data_stream.dataset"], "wslvault.audit") where log.attributes["appname"] == "wslvault"
          - set(log.attributes["data_stream.dataset"], "opsapi.sso") where log.attributes["appname"] == "opsapi-sso"
          - set(log.attributes["data_stream.dataset"], "cc.attestation") where log.attributes["appname"] == "cc-attest"
          - delete_key(log.attributes, "message")

  # Never let credentials or prompt text reach any backend
  transform/redact:
    error_mode: ignore
    log_statements:
      - statements:
          - delete_key(log.attributes, "http.request.header.authorization")
          - delete_key(log.attributes, "http.request.header.cookie")
          - delete_key(log.attributes, "gen_ai.prompt")
          - delete_key(log.attributes, "gen_ai.completion")
          - replace_pattern(log.body, "(?i)(bearer|api-token)\\s+[A-Za-z0-9._~+/=-]{16,}", "$$1 [REDACTED]") where IsString(log.body)
    trace_statements:
      - statements:
          - delete_key(span.attributes, "gen_ai.prompt")
          - delete_key(span.attributes, "gen_ai.completion")

  # Dynatrace wants delta sums; Prometheus wants cumulative ones
  cumulative_to_delta:
    max_staleness: 25h
  delta_to_cumulative:
    max_stale: 10m

  batch:
    send_batch_size: 4096
    timeout: 2s

connectors:
  # Everything goes to the SIEM; Medium and above is copied to Dynatrace
  routing/severity:
    default_pipelines: [logs/elastic]
    table:
      - condition: log.severity_number >= SEVERITY_NUMBER_WARN2
        action: copy
        pipelines: [logs/dynatrace]

  # Turn security events into low-cardinality counters for Prometheus
  count/security:
    logs:
      ws.security.events:
        description: Security events by source, outcome and action
        attributes:
          - key: data_stream.dataset
            default_value: unknown
          - key: event.outcome
            default_value: unknown
          - key: event.action
            default_value: unknown

exporters:
  elasticsearch/siem:
    endpoints:
      - https://es-sec-1.sec.internal:9200
      - https://es-sec-2.sec.internal:9200
    mapping:
      allowed_modes: [ecs]
    logs_dynamic_id:
      enabled: true
    logs_dynamic_pipeline:
      enabled: true
    auth:
      authenticator: basicauth/es
    tls:
      ca_file: /etc/otelcol-contrib/es-ca.pem
    sending_queue:
      enabled: true
      storage: file_storage

  otlp_http/dynatrace:
    endpoint: https://${env:DT_ENV_ID}.live.dynatrace.com/api/v2/otlp
    headers:
      Authorization: "Bearer ${env:DT_PLATFORM_TOKEN}"
    sending_queue:
      enabled: true
      storage: file_storage

  prometheus_remote_write/sec:
    endpoint: https://prometheus.sec.internal/api/v1/write
    # only the fleet labels; never sandbox or pod ids
    resource_constant_labels:
      included: [site, cluster]
    tls:
      ca_file: /etc/otelcol-contrib/ca.pem

service:
  extensions: [health_check, file_storage, basicauth/es]
  pipelines:
    logs/in:
      receivers: [otlp, tcp_log/argus, syslog/security]
      processors: [memory_limiter, resource/security, transform/ocsf, transform/appliances, transform/redact]
      exporters: [routing/severity, count/security]
    logs/elastic:
      receivers: [routing/severity]
      processors: []
      exporters: [elasticsearch/siem]
    logs/dynatrace:
      receivers: [routing/severity]
      processors: [batch]
      exporters: [otlp_http/dynatrace]
    metrics/scraped:
      receivers: [otlp, prometheus/security]
      processors: [memory_limiter, resource/security, batch]
      exporters: [prometheus_remote_write/sec]
    metrics/dynatrace:
      receivers: [otlp, prometheus/security]
      processors: [memory_limiter, resource/security, cumulative_to_delta, batch]
      exporters: [otlp_http/dynatrace]
    metrics/counts:
      receivers: [count/security]
      processors: [delta_to_cumulative, batch]
      exporters: [prometheus_remote_write/sec]
    metrics/counts-dt:
      receivers: [count/security]
      processors: [batch]
      exporters: [otlp_http/dynatrace]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource/security, transform/redact, batch]
      exporters: [otlp_http/dynatrace]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Notes on choices:

• Document ID from the OCSF event ID. OpenShell delivers at least once. Setting elasticsearch.document_id from ocsf.metadata.uid with logs_dynamic_id enabled makes a retried record a no-op: in our end-to-end test the second copy was rejected by Elasticsearch with a version conflict and only one document was stored.
• Dynamic ingest pipeline. The elasticsearch.ingest_pipeline attribute selects the OCSF pipeline only for OpenShell records; other datasets use the data stream's defaults.
• Routing by severity. The routing connector copies records at OTel severity WARN2 and above (OpenShell maps OCSF Medium to WARN2, High to ERROR) to Dynatrace, while every record goes to Elasticsearch. That keeps Dynatrace volumes and cost in proportion.
• Counting events into metrics. The count connector emits ws.security.events with only three attributes: dataset, outcome and action. It emits delta sums, which Dynatrace wants and Prometheus remote write does not, so the Prometheus path converts with delta_to_cumulative. In our test Prometheus received ws_security_events_total{data_stream_dataset="openshell.ocsf", event_action="Denied", event_outcome="failure"} with site and cluster labels.
• Cumulative to delta for Dynatrace. Dynatrace's OTLP ingest requires delta temporality for metrics, so scraped metrics pass through cumulative_to_delta on the Dynatrace path only.
• Constant labels. resource_constant_labels turns only site and cluster into Prometheus labels. Sandbox IDs, pod names and domains never become labels (section 14).
• Batching. The Elasticsearch exporter batches internally, so its pipeline has no batch processor; the Collector warns about double batching otherwise.

OpenShell side

On the OpenShell gateway, point OCSF export at the security gateways. NVIDIA's example uses an in-cluster service name; ours is the security gateway's address. Supervisors use the same endpoint, so it must be reachable from sandboxes, and they receive the gateway's export headers with their launch credentials rather than through the sandbox environment.

[openshell.gateway.otlp.ocsf]
enabled  = true
endpoint = "https://otel-security.sec.internal:4317"

Keep OpenShell's own anonymous product telemetry decision separate: it is unrelated to this export and is controlled by OPENSHELL_TELEMETRY_ENABLED or the Helm value server.telemetryEnabled.

7. Elastic Security as the SIEM

Elastic Security holds every security event, runs the detection rules, and is where analysts investigate (timelines) and track work (cases). We run it as a dedicated security cluster or a dedicated space with its own roles, separate from operational log search.

Ingest pipeline: OCSF to ECS

This pipeline was loaded into Elasticsearch 9.2.0 and exercised both with _simulate and with records arriving through the gateway.

PUT _ingest/pipeline/ws-ocsf-to-ecs
{
  "description": "OpenShell OCSF 1.8.0 records to ECS (Workstation)",
  "processors": [
    { "json": { "field": "message", "target_field": "ocsf", "if": "ctx.message instanceof String && ctx.message.startsWith('{')" } },
    { "set": { "field": "event.original", "copy_from": "message", "if": "ctx.ocsf != null" } },
    { "remove": { "field": "message", "if": "ctx.ocsf != null" } },
    { "date": { "field": "ocsf.time", "formats": ["UNIX_MS"], "target_field": "@timestamp", "if": "ctx.ocsf?.time != null" } },
    { "set": { "field": "event.module", "value": "openshell" } },
    { "set": { "field": "event.dataset", "value": "openshell.ocsf" } },
    { "set": { "field": "event.kind", "value": "event" } },
    { "set": { "field": "event.kind", "value": "alert", "if": "ctx.ocsf?.class_uid == 2004" } },
    { "set": { "field": "event.code", "value": "{{{ocsf.type_uid}}}", "if": "ctx.ocsf?.type_uid != null" } },
    { "set": { "field": "event.severity", "copy_from": "ocsf.severity_id", "if": "ctx.ocsf?.severity_id != null" } },
    { "set": { "field": "event.id", "copy_from": "ocsf.metadata.uid", "if": "ctx.ocsf?.metadata?.uid != null" } },
    { "set": { "field": "event.reason", "copy_from": "ocsf.status_detail", "if": "ctx.ocsf?.status_detail != null" } },
    { "set": { "field": "event.reason", "copy_from": "ocsf.finding_info.uid", "if": "ctx.ocsf?.finding_info?.uid != null" } },
    { "script": {
        "lang": "painless",
        "if": "ctx.ocsf?.class_uid != null",
        "source": "def m = ['4001': 'network', '4002': 'network', '4007': 'network', '1007': 'process', '3002': 'authentication', '6003': 'api', '5019': 'configuration', '2004': 'intrusion_detection', '6002': 'process']; def c = m.get(String.valueOf(ctx.ocsf.class_uid)); if (c != null) { if (ctx.event == null) { ctx.event = [:]; } ctx.event.category = [c]; }"
    } },
    { "set": { "field": "observer.product", "copy_from": "ocsf.metadata.product.name", "if": "ctx.ocsf?.metadata?.product?.name != null" } },
    { "set": { "field": "observer.version", "copy_from": "ocsf.metadata.product.version", "if": "ctx.ocsf?.metadata?.product?.version != null" } },
    { "set": { "field": "source.ip", "copy_from": "ocsf.src_endpoint.ip", "if": "ctx.ocsf?.src_endpoint?.ip != null" } },
    { "set": { "field": "destination.ip", "copy_from": "ocsf.dst_endpoint.ip", "if": "ctx.ocsf?.dst_endpoint?.ip != null" } },
    { "set": { "field": "destination.domain", "copy_from": "ocsf.dst_endpoint.domain", "override": false, "if": "ctx.ocsf?.dst_endpoint?.domain != null" } },
    { "set": { "field": "process.command_line", "copy_from": "ocsf.actor.process.cmd_line", "if": "ctx.ocsf?.actor?.process?.cmd_line != null" } },
    { "set": { "field": "process.pid", "copy_from": "ocsf.actor.process.pid", "if": "ctx.ocsf?.actor?.process?.pid != null" } },
    { "set": { "field": "user.id", "copy_from": "ocsf.actor.user.uid", "if": "ctx.ocsf?.actor?.user?.uid != null" } },
    { "set": { "field": "container.id", "copy_from": "ocsf.device.container.uid", "if": "ctx.ocsf?.device?.container?.uid != null" } },
    { "set": { "field": "container.name", "copy_from": "ocsf.device.container.name", "override": false, "if": "ctx.ocsf?.device?.container?.name != null" } },
    { "set": { "field": "url.full", "copy_from": "ocsf.http_request.url.url_string", "if": "ctx.ocsf?.http_request?.url?.url_string != null", "ignore_failure": true } },
    { "append": { "field": "related.hosts", "value": ["{{{destination.domain}}}"], "allow_duplicates": false, "if": "ctx.destination?.domain != null" } },
    { "append": { "field": "related.ip", "value": ["{{{source.ip}}}"], "allow_duplicates": false, "if": "ctx.source?.ip != null" } }
  ],
  "on_failure": [
    { "set": { "field": "event.kind", "value": "pipeline_error" } },
    { "append": { "field": "error.message", "value": ["{{{_ingest.on_failure_message}}}"] } }
  ]
}

Index template and retention

The ocsf object is mapped as flattened, so OCSF's many optional fields cannot cause a mapping explosion, but analysts can still filter on them. The original record is stored but not indexed. The template composes Elastic's built-in logs@mappings, logs@settings and ecs@mappings.

PUT _ilm/policy/ws-security-13m
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" } } },
      "warm":   { "min_age": "14d",  "actions": { "forcemerge": { "max_num_segments": 1 }, "allocate": { "number_of_replicas": 1 } } },
      "cold":   { "min_age": "90d",  "actions": { "set_priority": { "priority": 0 } } },
      "delete": { "min_age": "400d", "actions": { "delete": {} } }
    }
  }
}

PUT _index_template/logs-openshell.ocsf
{
  "index_patterns": ["logs-openshell.ocsf-*"],
  "data_stream": {},
  "priority": 200,
  "composed_of": ["logs@mappings", "logs@settings", "ecs@mappings"],
  "template": {
    "settings": { "index.lifecycle.name": "ws-security-13m" },
    "mappings": {
      "properties": {
        "ocsf": { "type": "flattened" },
        "event": { "properties": { "original": { "type": "keyword", "index": false, "doc_values": false } } }
      }
    }
  }
}

We create one template per security dataset with the same shape and a lifecycle policy that matches its retention requirement (section 14). Thirteen months is a common default for security events because it covers a full year of comparison plus the time to investigate an incident found late; your legal and compliance owners decide the real figure.

Detection rules

Fourteen rules, created through Kibana's detection engine API (POST /api/detection_engine/rules, API version 2023-10-31). Each one executed successfully against test data, and every rule except the out-of-hours vault rule (whose test event was inside hours) raised its expected alert. The table lists them all; two are shown in full after it.

The queries, as loaded:

# ws-ai-001-agent-egress-denied (KQL query)
data_stream.dataset:"openshell.ocsf" and event.category:"network" and event.action:"Denied"

# ws-ai-002-agent-new-egress-domain (New terms (KQL))
# new terms: destination.domain since now-14d
data_stream.dataset:"openshell.ocsf" and event.category:"network" and event.action:"Allowed" and destination.domain:*

# ws-ai-003-agent-ssrf-blocked (KQL query)
data_stream.dataset:"openshell.ocsf" and event.reason:"resolves to always-blocked address"

# ws-ai-004-sandbox-boundary-finding (KQL query)
data_stream.dataset:"openshell.ocsf" and event.kind:"alert" and event.reason:"sandbox-authorization-boundary"

# ws-ai-005-policy-denial-spike (Threshold (KQL))
# threshold: 25 or more per container.name
data_stream.dataset:"openshell.ocsf" and event.action:"Denied"

# ws-ai-006-guardrail-jailbreak-attempts (Threshold (KQL))
# threshold: 5 or more per user.id
data_stream.dataset:"nemo.guardrails" and event.action:"input_rail_blocked" and rule.name:"check jailbreak"

# ws-ai-007-garak-regression (ES|QL)
FROM logs-garak.eval-* METADATA _id, _index, _version
| WHERE garak.entry_type == "eval" AND garak.total_evaluated > 0
| EVAL attack_success_rate = garak.fails / garak.total_evaluated
| WHERE attack_success_rate > 0.05
| KEEP _id, _index, _version, @timestamp, ci.pipeline, garak.probe, garak.detector, garak.fails, garak.total_evaluated, attack_success_rate

# ws-ai-008-doca-argus-alert (KQL query)
data_stream.dataset:"doca.argus" and argus.message_type:"ALERT" and argus.severity:("HIGH" or "CRITICAL")

# ws-ai-009-sso-anomaly-then-agent-approval (EQL sequence)
sequence by user.id with maxspan=30m
  [authentication where data_stream.dataset == "opsapi.sso" and event.outcome == "success" and source.geo.country_iso_code != "GB"]
  [any where data_stream.dataset == "opsapi.sso" and event.action == "agent.approval.granted"]

# ws-ai-010-vault-read-out-of-hours (ES|QL)
FROM logs-wslvault.audit-* METADATA _id, _index, _version
| WHERE event.action == "secret.read" AND event.outcome == "success"
| EVAL hour = DATE_EXTRACT("hour_of_day", @timestamp)
| WHERE hour < 7 OR hour >= 19
| KEEP _id, _index, _version, @timestamp, user.id, wslvault.path, source.ip, hour

# ws-ai-011-k8s-privileged-pod (KQL query)
data_stream.dataset:"kubernetes.audit_logs" and kubernetes.audit.objectRef.resource:"pods" and kubernetes.audit.verb:("create" or "update" or "patch") and kubernetes.audit.requestObject.spec.containers.securityContext.privileged:true

# ws-ai-012-attestation-failure (KQL query)
data_stream.dataset:"cc.attestation" and event.outcome:"failure"

# ws-ai-013-waf-block-burst (Threshold (KQL))
# threshold: 50 or more per source.ip
data_stream.dataset:"wslproxy.waf" and event.action:"block"

# ws-ai-014-exec-from-tmp (KQL query)
data_stream.dataset:"auditd.log" and auditd.log.record_type:"SYSCALL" and auditd.data.syscall:"59" and auditd.data.exe:(/tmp/* or /dev/shm/*)

A query rule with alert suppression and an ATT&CK mapping. Suppression groups repeated denials for the same sandbox and destination into one alert per hour, which is what keeps this rule usable on a busy cluster:

{
  "rule_id": "ws-ai-001-agent-egress-denied",
  "name": "AI agent egress denied by OpenShell policy",
  "description": "An agent inside an OpenShell sandbox tried to reach a destination its policy does not allow. Often the first sign of indirect prompt injection or goal hijacking: the agent has been told to send data somewhere new.",
  "type": "query",
  "language": "kuery",
  "index": [
    "logs-openshell.ocsf-*"
  ],
  "query": "data_stream.dataset:\"openshell.ocsf\" and event.category:\"network\" and event.action:\"Denied\"",
  "severity": "medium",
  "risk_score": 47,
  "interval": "5m",
  "from": "now-6m",
  "alert_suppression": {
    "group_by": [
      "container.name",
      "destination.domain"
    ],
    "duration": {
      "value": 1,
      "unit": "h"
    },
    "missing_fields_strategy": "suppress"
  },
  "tags": [
    "Workstation",
    "Domain: AI agents",
    "Data Source: OpenShell"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0010",
        "name": "Exfiltration",
        "reference": "https://attack.mitre.org/tactics/TA0010/"
      },
      "technique": [
        {
          "id": "T1567",
          "name": "Exfiltration Over Web Service",
          "reference": "https://attack.mitre.org/techniques/T1567/"
        }
      ]
    }
  ]
}

An EQL sequence that correlates two sources an analyst would otherwise have to join by hand: a sign-in from outside the UK followed, within 30 minutes, by the same user approving privileged agent access:

{
  "rule_id": "ws-ai-009-sso-anomaly-then-agent-approval",
  "name": "Sign-in from outside the UK followed by an agent approval",
  "description": "An OpsAPI SSO sign-in from outside the UK, then the same user approving privileged agent access within 30 minutes.",
  "type": "eql",
  "language": "eql",
  "index": [
    "logs-opsapi.sso-*"
  ],
  "query": "sequence by user.id with maxspan=30m\n  [authentication where data_stream.dataset == \"opsapi.sso\" and event.outcome == \"success\" and source.geo.country_iso_code != \"GB\"]\n  [any where data_stream.dataset == \"opsapi.sso\" and event.action == \"agent.approval.granted\"]",
  "severity": "high",
  "risk_score": 73,
  "interval": "5m",
  "from": "now-35m",
  "tags": [
    "Workstation",
    "Domain: Identity",
    "Data Source: OpsAPI"
  ]
}

Three practical points about rules:

• ES|QL rules need metadata for deduplication. Non-aggregating ES|QL rules should select METADATA _id, _index, _version and keep those fields, as ours do, so the engine can avoid duplicate alerts.
• Use Elastic's prebuilt rules too. Elastic ships hundreds of prebuilt rules for Linux, Kubernetes, cloud and identity. Ours fill the AI-specific gaps. Where a prebuilt rule covers the same ground with a different data source (Elastic's own Kubernetes audit integration, for example), use one or the other, not both.
• Rules are code. We keep them in Git, load them in CI, and test each one with a crafted event, exactly as in this article's validation. A rule without a test is a guess.

Timelines and cases

Every alert opens into a timeline. For agent incidents we save a timeline template filtered on container.name (the sandbox) that pulls in the sandbox's OpenShell events, the guardrail events and garak results for the same application, and Argus alerts for the host it ran on, so an analyst sees the agent's last actions in order. Cases collect alerts, timelines, notes and response actions; we connect them to ServiceNow through Elastic's case connectors so the ticket and the case stay in step, and the response service (section 12) writes each automated action back to the case as a comment.

8. Dynatrace

Many enterprises run Dynatrace for application and infrastructure monitoring. Sending it the security signals that matter to service owners means a denied egress or a guardrail spike appears next to the service it affects, and Davis, Dynatrace's AI engine, can relate it to changes in that service's behaviour.

• Endpoint. Per Dynatrace's OTLP API documentation, the base URL for SaaS is https://{your-environment-id}.live.dynatrace.com/api/v2/otlp (or an ActiveGate address), and the exporter appends /v1/traces, /v1/metrics and /v1/logs. Our test Collector's own metrics confirmed it was posting to /api/v2/otlp/v1/logs.
• Protocol. OTLP over HTTP with binary protobuf only: no gRPC and no JSON. Hence otlp_http in the gateway.
• Authentication. A Dynatrace platform token in an Authorization: Bearer header, with the openpipeline:logs:ingest, openpipeline:metrics:ingest and openpipeline:traces:ingest scopes. Older environments using classic access tokens should follow Dynatrace's documentation for their token type.
• Metric temporality. Delta only, as noted above.

What we send: OpenShell and other events at Medium severity and above, the ws.security.events counters, the scraped OpenShell and WAF metrics, and guardrail traces (with content capture off). What we do in Dynatrace, per its documentation rather than any API we invent here: set up anomaly detection on the security counters so Davis learns their normal level per cluster and raises a problem on a sustained change; add the security events to the dashboards and notebooks that service owners already use; and, where the customer's Dynatrace licence includes its security features, route the high-severity records into Dynatrace's security event storage through OpenPipeline so they can be analysed alongside vulnerability findings. We have not verified OpenPipeline routing rules in a live tenant for this article, so we describe that step without configuration.

Dynatrace is a second opinion and a service-owner view, not a second SIEM. Detection, investigation and cases stay in Elastic Security, so there is one place where an incident is owned.

9. Prometheus and Alertmanager

Prometheus is the fastest and cheapest place to alert on numbers. Three security use cases are naturally numeric, and the data for two of them (GPU utilisation from DCGM and network bytes from node-exporter) is already in the operations metrics store from the UFM design. The third uses the event counters the security gateway produces. The second group of rules watches the security pipeline itself.

# /etc/prometheus/rules/ai-security.yml
groups:
  - name: ai-security-signals
    rules:
      # GPU busy with no Kubernetes pod and no Slurm job attached: possible cryptojacking.
      # Needs dcgm-exporter's pod mapping (Kubernetes) or HPC job mapping (Slurm) switched on.
      - alert: GpuBusyWithoutScheduledWork
        expr: |
          max by (site, cluster, Hostname, gpu) (ws:gpu_util:ratio{pod="", hpc_job=""}) > 0.3
        for: 15m
        labels:
          severity: high
          team: soc
          use_case: cryptojacking
        annotations:
          summary: "GPU {{ $labels.gpu }} on {{ $labels.Hostname }} busy with no scheduled job"
          description: "Utilisation {{ $value | humanizePercentage }} for 15 minutes with no pod or Slurm job label. Check for miners and unscheduled processes."

      # Egress from inference and storage nodes far above the same hour last week:
      # model weights or datasets leaving.
      - alert: UnusualEgressFromModelNodes
        expr: |
          (
            sum by (site, cluster, instance) (increase(node_network_transmit_bytes_total{node_role=~"inference|storage", device!~"lo|ib.*"}[1h]))
              > 3 * sum by (site, cluster, instance) (increase(node_network_transmit_bytes_total{node_role=~"inference|storage", device!~"lo|ib.*"}[1h] offset 7d))
          )
          and
          sum by (site, cluster, instance) (increase(node_network_transmit_bytes_total{node_role=~"inference|storage", device!~"lo|ib.*"}[1h])) > 50e9
        for: 10m
        labels:
          severity: high
          team: soc
          use_case: exfiltration
        annotations:
          summary: "{{ $labels.instance }} sent {{ $value | humanize1024 }}B in the last hour, over 3x last week"

      # OpenShell denials across the cluster, from the gateway's count connector
      - alert: AgentPolicyDenialSurge
        expr: |
          sum by (site, cluster) (rate(ws_security_events_total{data_stream_dataset="openshell.ocsf", event_action="Denied"}[10m]))
            > 3 * sum by (site, cluster) (rate(ws_security_events_total{data_stream_dataset="openshell.ocsf", event_action="Denied"}[1d] offset 1d))
          and
          sum by (site, cluster) (rate(ws_security_events_total{data_stream_dataset="openshell.ocsf", event_action="Denied"}[10m])) > 0.2
        for: 10m
        labels:
          severity: medium
          team: soc
          use_case: agent-misuse
        annotations:
          summary: "OpenShell denials in {{ $labels.cluster }} at 3x yesterday's rate"

  - name: ai-security-pipeline-health
    rules:
      # Losing security telemetry is itself a security event
      - alert: OpenShellOcsfExportDropping
        expr: sum by (namespace, pod) (increase(openshell_ocsf_otlp_dropped_total[10m])) > 0
        labels:
          severity: high
          team: soc
        annotations:
          summary: "OpenShell gateway {{ $labels.pod }} dropped OCSF audit records"

      - alert: OpenShellInterceptorFailOpen
        expr: sum by (namespace, pod) (increase(openshell_gateway_interceptor_fail_open_total[10m])) > 0
        labels:
          severity: high
          team: soc
        annotations:
          summary: "OpenShell interceptor failed open on {{ $labels.pod }}"

      - alert: WafFailingOpen
        expr: sum by (host) (rate(waf_errors[5m])) > 0
        for: 10m
        labels:
          severity: medium
          team: platform
        annotations:
          summary: "wslproxy WAF on {{ $labels.host }} is erroring and therefore allowing requests"

      - alert: SiemExportFailing
        expr: |
          sum by (instance, exporter) (rate({__name__=~"otelcol_exporter_send_failed_log_records(_total)?", exporter="elasticsearch/siem"}[5m])) > 0
        for: 10m
        labels:
          severity: high
          team: soc
        annotations:
          summary: "Security gateway {{ $labels.instance }} cannot write to the SIEM"

      - alert: SecurityGatewayRedundancyLost
        expr: count by (site) (up{job="otel-security-gateway"} == 1) < 2
        for: 5m
        labels:
          severity: high
          team: soc
        annotations:
          summary: "Only one security gateway left at {{ $labels.site }}"

Notes:

• ws:gpu_util:ratio is the recording rule from the UFM article that bridges dcgm-exporter's old and new metric names. The cryptojacking rule relies on dcgm-exporter attaching pod (Kubernetes) or hpc_job (Slurm) to GPUs that are in use. If job mapping is not switched on, every busy GPU will match, so check that first.
• node_role is the label the UFM agent sets from WS_NODE_ROLE; use whatever role label your fleet has.
• The Collector's own metric names lost their _total suffix in the 0.165.0 build we tested (otelcol_exporter_send_failed_log_records), so SiemExportFailing matches both spellings.
• Rules that compare with last week or yesterday need that much history before they mean anything. Run them in a non-paging route for their first two weeks.

Unit tests for the two rules most likely to be edited, run with promtool test rules:

rule_files:
  - security-rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      # GPU 0 busy, no job: should fire.  GPU 1 busy inside a pod: must not.
      - series: 'ws:gpu_util:ratio{site="lon2",cluster="ai-prod-1",Hostname="gpu-r07-n03",gpu="0"}'
        values: '0.95x30'
      - series: 'ws:gpu_util:ratio{site="lon2",cluster="ai-prod-1",Hostname="gpu-r07-n03",gpu="1",pod="train-7f9",namespace="research"}'
        values: '0.95x30'
    alert_rule_test:
      - eval_time: 20m
        alertname: GpuBusyWithoutScheduledWork
        exp_alerts:
          - exp_labels:
              severity: high
              team: soc
              use_case: cryptojacking
              site: lon2
              cluster: ai-prod-1
              Hostname: gpu-r07-n03
              gpu: "0"
            exp_annotations:
              summary: "GPU 0 on gpu-r07-n03 busy with no scheduled job"
              description: "Utilisation 95% for 15 minutes with no pod or Slurm job label. Check for miners and unscheduled processes."
  - interval: 1m
    input_series:
      - series: 'openshell_ocsf_otlp_dropped_total{namespace="openshell",pod="openshell-0",reason="queue_capacity"}'
        values: '0 0 0 5 9 9 9 9 9 9 9 9'
    alert_rule_test:
      - eval_time: 8m
        alertname: OpenShellOcsfExportDropping
        exp_alerts:
          - exp_labels:
              severity: high
              team: soc
              namespace: openshell
              pod: openshell-0
            exp_annotations:
              summary: "OpenShell gateway openshell-0 dropped OCSF audit records"

Alertmanager sends high and critical security alerts to PagerDuty and to the response service, and everything else to tickets. Secrets come from files, never inline:

# /etc/alertmanager/alertmanager.yml  (security routes only)
route:
  receiver: platform-tickets
  group_by: [alertname, site, cluster]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: ['team="soc"', 'severity=~"high|critical"']
      receiver: soc-page-and-respond
      continue: true
    - matchers: ['team="soc"']
      receiver: soc-tickets

inhibit_rules:
  # If the whole security gateway tier is down, do not also page for every export failure
  - source_matchers: ['alertname="SecurityGatewayRedundancyLost"']
    target_matchers: ['alertname="SiemExportFailing"']
    equal: [site]

receivers:
  - name: soc-page-and-respond
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-soc
        severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}error{{ end }}'
    webhook_configs:
      # Workstation response service: opens the case and runs the matching playbook
      - url_file: /etc/alertmanager/secrets/soar-webhook-url
        send_resolved: true
  - name: soc-tickets
    webhook_configs:
      - url_file: /etc/alertmanager/secrets/servicenow-bridge-url
        send_resolved: true
  - name: platform-tickets
    webhook_configs:
      - url_file: /etc/alertmanager/secrets/servicenow-bridge-url

Alertmanager has no native ServiceNow receiver; the webhook goes to a small bridge or to ServiceNow's own event integration, depending on what the customer runs.

10. Grafana SOC dashboards

Grafana is the shared screen for the SOC, platform and AI teams, with Prometheus and Elasticsearch as data sources (and Loki only if a customer wants Grafana-native log search for non-security logs; security events stay in Elasticsearch). One overview dashboard, with drill-downs:

Dashboards are provisioned from Git like the rules. We give each panel a description that says what normal looks like and which rule or runbook applies, so a new analyst can use the screen on day one.

11. Detection use cases

Each use case pairs a signal with a rule and a planned response. Response levels are explained in section 12: auto means a pre-approved, reversible action; approve means one click by a named approver; advise means the case recommends and people act.

12. Response and SOAR

Detection without response is a report. We wire responses to alerts through a small response service (the webhook target in Alertmanager, and an Elastic connector for detection rules) that opens or updates the case, checks the response level for the rule, and either runs a playbook, asks for approval, or only records a recommendation.

Containment options, from narrowest to widest

• Stop the sandbox (OpenShell). openshell sandbox stop <name> stops an agent and keeps the sandbox for investigation; openshell sandbox delete <name> removes it once evidence is collected. To withdraw one permission rather than stop the agent, openshell policy update <name> --remove-rule <rule> --wait. Collect the supervisor's OCSF files before deletion: on Kubernetes they live in the supervisor pod's emptyDir and disappear with it.
• Quarantine the pod (Kubernetes). A namespace-wide NetworkPolicy isolates any pod carrying a quarantine label; the playbook only adds the label, which is fast, reversible and easy to audit. Your CNI must enforce NetworkPolicy, as OpenShell's own Kubernetes guidance also requires.
• Revoke secrets. Revoke the agent identity's leases in the key service so anything it exfiltrated stops working. With wslvault the agent never held long-lived keys in the first place, which limits what there is to revoke.
• Isolate the host (BlueField and Sentry). Where Sentry on BlueField-4 is deployed, NVIDIA's design quarantines an agent at the DPU in milliseconds, independently of the host. Where it is not, the fallback is to cordon and drain the node and apply a host-level network block. We drive DPU-side actions only through the interfaces NVIDIA documents for the deployed release; we do not script undocumented ones.
• Escalate to the XDR platform. Where DOCA Argus feeds a customer's XDR, that platform's own response actions remain available and the case records which tool acted.

# Applied once per namespace; the playbook only adds the label
#   kubectl -n agents label pod <pod> ws.security/quarantine=true
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
  namespace: agents
spec:
  podSelector:
    matchLabels:
      ws.security/quarantine: "true"
  policyTypes: [Ingress, Egress]
  ingress:
    # forensics tooling only
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: soc-forensics
  egress:
    # DNS stays up so the pod's own logs about failures still make sense
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Approval gates through OpsAPI single sign-on

Every response that is not pre-approved goes through an approval step in OpsAPI: the approver signs in with single sign-on (and multi-factor authentication), sees the case summary, and approves or rejects. The approval is itself a security event (opsapi.sso, agent.approval.granted and its siblings), which is why use case 11 can watch for a suspicious sign-in followed by an approval. We configure it so that approvers cannot approve their own requests, and changes to the security pipeline's own configuration need two people. The same pattern is described for agent permissions in our agentic AI security article.

ServiceNow and PagerDuty

High and critical alerts page through PagerDuty; everything opens or updates a ServiceNow incident (or security incident, if the customer runs that module), linked to the Elastic case. Resolution flows back: when the case closes, the incident resolves and the PagerDuty alert clears. We agree the automation level for each rule with the customer in writing, review it quarterly, and only promote a playbook to auto after a clean record at approve.

13. Compliance mapping

Monitoring is how you show that controls work. The mapping below is general guidance, not legal advice; your data protection officer and compliance owners make the final calls.

14. Retention, cost and cardinality

Security telemetry grows quietly until the bill arrives. Four rules keep it predictable:

1. Retain by dataset, not globally. A typical starting point: OpenShell, SSO, vault, Kubernetes audit, attestation and Argus at 13 months; WAF decisions at 90 days in Elasticsearch with longer archive if required; garak evaluation records for the life of the model; allowed (not denied) OpenShell network events at 30 to 90 days, because they are mostly useful for baselines. Older data can move to searchable snapshots or object storage if the Elastic licence allows.
2. Size from events per agent. As a worked example, not a measurement: 200 agents producing an average of 2,000 OCSF events an hour at about 1.2 KB each is roughly 11.5 GB a day before replicas and compression. Allowed network events dominate; denials and findings are a small fraction. Measure your own rate in a pilot before sizing.
3. Metrics stay low-cardinality. The event counter has three labels with a handful of values each, plus site and cluster. Sandbox IDs, pod names, user IDs and domains are unbounded and belong in Elasticsearch, never in Prometheus labels. One mis-chosen label (a sandbox ID on a busy cluster) can add thousands of new series a day.
4. Send each backend only what it uses. Every event goes to Elasticsearch once; Dynatrace gets Medium and above; Prometheus gets counts. Traces stay sampled at the application. Duplicating everything into every tool is the most common cost mistake we see.

15. High availability of the security pipeline

An attacker who can blind the SIEM has won half the battle, so the pipeline is built to keep working through failures and to shout when it cannot.

• Gateways: at least two per site behind a load balancer or DNS name, stateless apart from their persistent queues on local disk. Agents and OpenShell retry to whichever gateway answers; a gateway restart loses nothing queued.
• At-least-once end to end: persistent sending queues on agents and gateways, OpenShell's own retries, and document IDs from event IDs so retries never duplicate.
• Elasticsearch: at least three master-eligible nodes across failure domains, one replica for every security index, snapshot to a separate store, and a write role for the gateways that cannot delete data.
• Prometheus and Alertmanager: two Prometheus replicas (deduplicated in Mimir, as in the UFM design) and a three-node Alertmanager cluster.
• Self-monitoring: the pipeline-health rules in section 9, plus a heartbeat: a synthetic OpenShell denial generated every five minutes in a canary sandbox must arrive in Elasticsearch, or a rule pages. Gap records from OpenShell (openshell.ocsf.gap) are searchable events too.
• Separation of duties: the security team owns the gateway configuration and the SIEM; changes go through review by two people; the platform team can see pipeline health but cannot change what is collected.
• Clocks: every source and gateway synchronised to the same time sources, and @timestamp taken from the source event, not ingest time, so sequences across sources line up.

16. Proving it works: drills at handover

We do not hand over a detection we have not seen fire. With the customer's team watching, we run and record:

1. An agent in a test sandbox reads a page with a planted instruction and tries to upload to a paste site: one OpenShell denial, one suppressed Elastic alert, a timeline showing the read and the attempt.
2. A test sandbox calls another sandbox's API with its own credential: a critical finding, the automatic stop, the quarantine label applied, the case updated with each action.
3. A known-bad probe is added to a CI branch: garak's attack success rate crosses 5% and the pipeline gate fails.
4. A benchmark job runs on a GPU with no scheduler job attached: the cryptojacking alert fires within its 15-minute window.
5. A test pod with a privileged container is created: the Kubernetes rule fires and the playbook asks for approval.
6. One security gateway is stopped: no lost events, a redundancy page, recovery when it returns.
7. The SIEM is blocked from the gateways for 15 minutes: the export-failure page fires, queues fill and drain afterwards, and the heartbeat shows no gap.

17. How Workstation delivers and runs this

1. Assessment. Inventory the estate, the agents and what they can touch, the data and model assets, and the existing SIEM, monitoring and response tools. Deliverable: a threat and control map like section 2, specific to you.
2. Architecture. Choose sources, schema, backends, retention, HA and the UK GDPR position. Deliverable: a signed design.
3. Build. Collector agents and security gateways, OpenShell OCSF export, ingest pipelines, index templates, lifecycle policies and dashboards, all from version-controlled configuration in your repositories.
4. Detection engineering. Rules, alerts and playbooks for your use cases, each with a test event and an owner.
5. Tuning. Run quietly for two to four weeks, cut noise, set thresholds and suppression, and agree an alert budget the team can sustain.
6. Handover. The drills in section 16, runbooks, and training for analysts and platform engineers.
7. Managed monitoring support. Ongoing monitoring support alongside your team (or as your team's out-of-hours cover): rule updates as NVIDIA's products and the threat landscape change, upgrades of the pipeline, regular threat-hunting reviews, and a monthly report on detections, response times, noise and coverage. The exact service levels are agreed per customer.

NVIDIA partner: engineering, software and people

Workstation is an NVIDIA partner, so alongside the hardware we provide the engineering, software and staff to set up and run AI data centres end to end, including the security monitoring around them. We specify and supply the platform, install and configure it, build the observability and SIEM integration described here, and provide engineers who work inside your change process afterwards.

• DPUs and networking: NVIDIA BlueField DPUs (including the BlueField-3 B3220, the minimum generation for DOCA Argus), ConnectX-8 SuperNICs and the Spectrum-X Ethernet platform.
• GPU systems: DGX B300, HGX B300 servers, DGX SuperPOD and Vera Rubin NVL72, the system class in NVIDIA's OpenShell Reference Architecture for inference.
• Software: NVIDIA AI Enterprise subscriptions, and set-up and support for OpenShell, NeMo Guardrails, garak and the open components in this article.

To talk through your estate, ask our AI Sales Engineer and it will route you to the right engineer.

18. Sources and verification notes

• NVIDIA portfolio overview: the official NVIDIA cybersecurity AI solutions page, read in November 2026, for OpenShell and its reference architecture, Sentry on BlueField, BlueField and DOCA, DOCA Argus, DOCA Vault, confidential computing, Nemotron and Nemotron Safety, NeMo, garak and NemoClaw. All wording here is our own summary.
• OpenShell: the GitHub repository README and its documentation pages on sandbox logging, OCSF JSON export, gateway configuration (OCSF JSONL and OTLP export), gateway metrics, sandboxes and policies. OCSF classes, reason strings, the sandbox-authorization-boundary finding, OTLP attributes, metric names and CLI commands are quoted from those pages. The sample OCSF record in our test was built from the documentation's example.
• DOCA Argus: NVIDIA's DOCA Argus service guide for requirements, message types, severities and telemetry keys. Not tested on hardware; field paths should be checked against a real message from your release.
• Sentry, DOCA Vault and the OpenShell Reference Architecture: described from NVIDIA's public page only. No interfaces, formats or commands for them are given here.
• Confidential computing: NVIDIA's page and the nvTrust README. The attestation event format is Workstation's convention.
• NeMo Guardrails: the repository README (rail types, release 0.24.1) and its tracing configuration reference. The block-event convention is Workstation's.
• garak: the repository README, CLI options and evaluator source for the eval record fields. The probe set in the CI example is illustrative.
• Morpheus: the repository README, examples and release list (latest tag 25.06, July 2025, at the time of writing).
• Dynatrace: Dynatrace's OTLP API and Collector configuration documentation for the endpoint, protocol, token scopes and delta temporality. Not tested against a live tenant: the exporter was validated and ran, but its requests went to a placeholder environment. Davis anomaly detection and security-event routing are described generally, per Dynatrace's documentation.
• OpenTelemetry Collector: both configurations validated with otelcol-contrib validate in otel/opentelemetry-collector-contrib:0.165.0, then run: test OCSF, Argus, appliance, guardrail, auditd, Kubernetes audit and garak records were sent through an agent and the gateway into Elasticsearch and Prometheus, and the resulting documents and series were inspected. Endpoints and hostnames are placeholders.
• Elastic: ingest pipeline, index template and lifecycle policy applied to Elasticsearch 9.2.0; all fourteen rules created through Kibana 9.2.0's detection engine API, executed successfully, and raised the expected alerts (thirteen fired; the out-of-hours rule correctly did not, because its test event was inside hours). Timeline templates, case connectors and prebuilt rule overlap were not tested.
• Prometheus and Alertmanager: rules validated with promtool check rules and unit-tested with promtool test rules, Prometheus 3.12.0; Alertmanager configuration validated with amtool check-config, Alertmanager 0.34.2. DCGM metric names follow dcgm-exporter 4.9.0 through the UFM article's ws: recording rules.
• Kubernetes: the quarantine NetworkPolicy validated with kubeconform 0.8.0 in strict mode. Not tested against a live CNI.
• Workstation conventions: dataset names, rule IDs, the ws.security.events metric, action names for wslvault and OpsAPI, the guardrail block event and the attestation event are ours.
• Illustrative figures: event rates, volumes, thresholds and timings are worked examples, not measurements from a customer system.
• Compliance: general guidance only, not legal advice.

Acknowledgement of rights: NVIDIA, the NVIDIA logo, BlueField, DOCA, DOCA Argus, DOCA Vault, Sentry, OpenShell, NemoClaw, Nemotron, NeMo, NeMo Guardrails, Morpheus, Vera, Vera Rubin, Blackwell, DGX, HGX, Spectrum, Spectrum-X, ConnectX, SuperNIC, NVLink, DCGM, UFM and NVIDIA AI Enterprise are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. NVIDIA products and technologies may be covered by patents and other intellectual property rights owned by NVIDIA Corporation, and all such rights are acknowledged and reserved to their owners. Product information is summarised in our own words from NVIDIA's public material; NVIDIA's website and documentation are the authoritative source. garak is an open-source project maintained by NVIDIA. Elastic, Elasticsearch, Kibana, Logstash, Elastic Security, Dynatrace, Davis, Prometheus, Alertmanager, Grafana, Loki, OpenTelemetry, Kubernetes, ServiceNow, PagerDuty, MITRE ATT&CK, Fluent Bit, Vector and Slurm are trademarks or registered trademarks of their respective owners, and other names may be trademarks of their respective owners. No endorsement by any of these companies is implied. Workstation's own configurations, rules, diagrams and text are © Workstation.