NVIDIA UFM · InfiniBand · OpenTelemetry

See the fabric before it costs you GPU hours

One marginal cable can slow every job that crosses it. This companion to our blog post and technote shows what we monitor in an NVIDIA AI cluster: the InfiniBand and Ethernet fabrics through UFM and NetQ, the GPUs through DCGM, the workloads through Slurm, Kubernetes and OpenTelemetry traces, and the monitoring stack itself.

For IT directors and CTOsWhy fabric visibility drives GPU utilisation and cost, the three UFM tiers, three cluster examples and what Workstation does.Read the blog For platform and HPC engineersEvery UFM SDK plugin, DCGM and the dcgm-exporter 4.9.0 renames, validated OpenTelemetry configuration, alert rules and the HA design.Read the technote Seventeen example dashboardsOpen each one to see a mock panel, the PromQL, KQL or TraceQL behind it and the alert thresholds we start from.Open the gallery

UFM tiers at a glance

Each tier includes the one before it. We confirm current packaging and licensing with NVIDIA for every quote.

UFM TelemetrySee the fabric.

Port, error and cable counters
Streams to Prometheus, Fluentd and others
Fits small, single-team clusters

UFM EnterpriseRun the fabric.

Topology, subnet manager, partitions
Events, alarms, REST API, HA pair
Slurm integration, SDK plugins

UFM Cyber-AIPredict the fabric.

Learns normal behaviour per fabric
Predicts failing links and cables
Fits large and multi-tenant fabrics

Four things to remember

Label everything the same way. Site, cluster, fabric, rack, node and rail on every signal, or no query will ever join a job to a switch port.
Keep job IDs off fabric series. Put them on GPU metrics and join through a small info metric.
Check your dcgm-exporter version. 4.9.0 renamed the default metrics and changed units; every example here uses the new names.
Make the monitoring highly available. UFM pair, two gateways, Mimir across three zones, three Elasticsearch masters, and a dead man's switch.

Ready to size a cluster? Ask our AI Sales Engineer or browse networking and DGX systems in the shop.

All dashboard values on this page are illustrative example data, not measurements from a customer system.

Blog · for IT directors, CTOs and research-computing heads

NVIDIA UFM for AI Clusters: Why InfiniBand Fabric Visibility Decides What Your GPUs Are Worth

The GPUs get the attention and the budget. The network that joins them decides how much of that budget turns into useful work. This post is for IT directors, CTOs and heads of research computing running or buying an NVIDIA InfiniBand cluster. It explains, in plain terms, what NVIDIA Unified Fabric Manager (UFM) does, which of its three tiers fits which cluster, what better fabric visibility is worth in GPU hours, and how Workstation supplies, sets up and runs the monitoring for you. Engineers who want the plugins, the OpenTelemetry configuration and the alert rules should read the technical article.

An InfiniBand fabric of spine and leaf switches joining GPU nodes, with one flapping link highlighted and a fabric-health dashboard beside it.

In one sentence. In a GPU cluster, one bad cable can slow every job that crosses it, and you only see that if someone is watching the fabric as closely as the GPUs; UFM is NVIDIA's tool for doing that on InfiniBand, and the right tier depends on the size of the cluster and how many teams share it.

Why the fabric decides what your GPUs are worth

Training a large model, or serving one across several servers, is a team sport for GPUs. Every few milliseconds they stop and exchange results with each other over the network. The job can only move as fast as the slowest exchange. If one link out of several hundred is running with errors, retrying packets or quietly dropping to a lower speed, every GPU in that job waits for it.

That is what makes AI networking different from office networking. On a normal network, a slow link annoys a few users. On an AI fabric, a slow link turns a 256-GPU job into a 256-GPU queue. And the symptom does not look like a network problem. It looks like "training is slower this week", or "the GPUs are only at 60%". Teams then spend days tuning software that was never the cause.

The maths is simple enough to put in front of a finance director. Take a cluster of 256 GPUs and give each GPU-hour a fully loaded cost of £2.50 (hardware depreciation, power, cooling, space and support; your figure will differ). That cluster represents about 2.24 million GPU-hours a year:

Losing 1% of utilisation to fabric problems costs about £56,000 a year.
One flapping link that stalls a 256-GPU training run for three hours before anyone notices costs about 770 GPU-hours, or £1,900, plus the restart from the last checkpoint.
A failing transceiver replaced in a planned window instead of mid-run saves the same again, and the weekend call-out.

These are illustrative figures, but the shape is always the same: the cost of watching the fabric properly is small compared with the GPU time it protects.

What NVIDIA UFM is, in plain terms

InfiniBand is the network most large AI and HPC clusters use between GPU servers, because it is fast, predictable and offloads a lot of work from the servers. Every InfiniBand fabric needs a subnet manager, the software that discovers every switch and adapter, works out routes and keeps the fabric running. NVIDIA Unified Fabric Manager is NVIDIA's software for running and watching that fabric at data-centre scale.

Think of it as three jobs in one product family: NVIDIA describes the full platform on its official UFM page.

See it. Collect counters from every switch port, adapter and cable: traffic, errors, retries, temperatures, link speeds. Show them live and keep history.
Run it. Discover the topology, manage the subnet manager, carve the fabric into isolated partitions for different teams or customers, apply firmware and configuration consistently, and raise events and alarms.
Predict it. Learn what normal looks like for your fabric and flag the cable that is getting worse, or the traffic pattern that does not fit, before it fails.

UFM also has an open software development kit with plugins that send its data to the tools your operations team already uses: Prometheus and Grafana, Elasticsearch via Fluentd, Zabbix, syslog, and the Slurm job scheduler. That matters, because nobody wants another screen to watch. We cover the plugins in detail in the technical article.

The three UFM tiers and when each fits

NVIDIA packages UFM in three main tiers. The names describe what each adds:

Tier	What it adds	Fits best when	Watch out for
UFM Telemetry	Real-time collection of fabric counters and cable data, streamed out to your own monitoring tools. Visibility only.	Smaller clusters where the subnet manager already runs on a switch, or teams that just need fabric data in Grafana.	No partition management, no central fabric control and no high-availability pair.
UFM Enterprise	Everything in Telemetry, plus full fabric management: discovery, topology views, subnet manager control, partitions, events and alarms, health reports, a REST API, scheduler integration and an active/standby HA pair.	Most production AI clusters, anything shared between teams or tenants, and any cluster running Slurm.	Needs dedicated management servers (or NVIDIA's appliance), and someone who owns it.
UFM Cyber-AI	Analytics on top of Enterprise telemetry history: learns each fabric's normal behaviour, predicts failing links and cables, and flags anomalies in performance or usage.	Large fabrics with thousands of cables, where predicting failures pays for itself, and providers selling GPU capacity who need to spot misuse.	Its value grows with fabric size and history; on a single-switch cluster it is usually more than you need.

Exact features, licensing and appliance options change between UFM releases, so we confirm the current packaging with NVIDIA for every quote. The decision logic, though, is stable: Telemetry if you only need to see the fabric, Enterprise if you need to run it, Cyber-AI if the fabric is big enough that predicting failures saves real money.

Three clusters, three answers

1. A four-node DGX B300 or HGX B300 cluster

Four eight-GPU servers, 32 GPUs, used by one AI team for fine-tuning and inference. Each server has eight 800 Gb/s network adapters for the GPU fabric, so there are 32 compute ports, which fit on a single InfiniBand switch.

Our usual recommendation: run the subnet manager on the switch and use UFM Telemetry to feed fabric counters into the same Grafana that shows GPU utilisation. If the cluster will later grow or be shared, start with UFM Enterprise so the partitioning and history are already in place.
Business outcome: one dashboard shows GPUs and network together. When a job slows down, the team can see within minutes whether a GPU, a server or a port is to blame, instead of opening a support case and waiting.
Example: a fine-tuning job's throughput dips by a fifth every afternoon. The fabric panel shows rising error-recovery counts on one port as the room warms up. The cause is a transceiver that is sensitive to temperature; it is swapped under warranty before it fails outright.

2. A 32-node SuperPOD-style cluster

Thirty-two eight-GPU servers, 256 GPUs, built on NVIDIA's reference design with separate network rails, leaf and spine switches and well over a thousand optical links. Several teams train large models on it, and some jobs use every GPU at once.

Our usual recommendation: UFM Enterprise in an HA pair, with partitions per team, the problem-detection plugin enabled first in dry-run mode, and the telemetry flowing into central monitoring. Evaluate UFM Cyber-AI once there are a few months of history: at this size, a handful of failing cables a quarter is normal, and predicting them is worth money.
Business outcome: faults that used to surface as "the big job is slow" become a ticket that says which cable, which rack and which jobs crossed it. Maintenance moves from emergency to scheduled.
Example: a link between a leaf and a spine starts to flap twice an hour. Alerting pages the on-call engineer in about five minutes, UFM's problem-detection plugin isolates the port so traffic routes around it, and the 256-GPU job carries on at a fraction of a percent lower bandwidth instead of stalling. The cable is replaced the next morning.

3. A university HPC service with Slurm

A shared research cluster: a few generations of GPU nodes, a large pool of CPU nodes, a parallel file system and hundreds of users across departments, all scheduled through Slurm.

Our usual recommendation: UFM Enterprise with NVIDIA's Slurm integration, which can give each job its own fabric partition while it runs, so jobs from different groups are isolated on the network as well as on the servers. GPU metrics are tagged with the Slurm job ID, so usage can be traced to a job, a user and a department.
Business outcome: fair, evidence-based reporting. The research computing team can show each department its GPU hours, how efficiently they were used and whether slow jobs were the code or the infrastructure. That makes funding bids, chargeback and capacity planning much easier.
Example: a PhD student reports that their eight-node job is "always slow on Tuesdays". The job view shows their nodes share a leaf switch with a weekly data-ingest job that saturates its uplinks. The fix is a scheduler constraint, not new hardware.

What good fabric visibility gives you

Higher GPU utilisation, because network faults are found in minutes, not days, and jobs are not left waiting on one bad port.
Fewer failed long runs, because failing cables and transceivers are spotted and replaced in planned windows.
Faster, calmer incidents, because GPU, server, network and job data sit on one timeline, so the first question ("is it the network?") gets answered with evidence.
Clean multi-tenancy, because partitions and per-job isolation are managed and audited centrally.
Honest capacity and cost reporting, because you can show cost per useful GPU-hour, by team, rather than cost per installed GPU.
Stronger warranty and support cases, because you can hand the vendor the counters, timestamps and history for a faulty part instead of a description.

Monitor the whole stack, not just the fabric

UFM covers the InfiniBand fabric. A working AI cluster also needs visibility of the GPUs (NVIDIA's Data Center GPU Manager, DCGM), the Ethernet networks for storage and front-end traffic (NVIDIA Spectrum-X and NetQ), BlueField data processing units if you use them, the servers, the scheduler and the AI services themselves. In our earlier post on observability as a business standard we argued that monitoring should be one company-wide contract, not a set of team preferences. The same applies here: fabric data belongs in the same system as everything else, labelled the same way, so a single query can follow a slow job from the scheduler to the GPU to the switch port.

Our standard design uses OpenTelemetry to collect everything once and send it to two kinds of backend: Prometheus-compatible metrics with Grafana (we use Grafana Mimir for long-term, highly available storage, and Grafana Tempo for request traces), and the Elastic stack (Elasticsearch and Kibana) for logs and events, because many organisations already run it. If you have already standardised on one of them, we plug into what you have.

What Workstation does

We are a UK reseller and integrator of NVIDIA AI infrastructure, and we run managed observability for customers. For a UFM-managed cluster that means three things:

Supply. GPU servers (DGX and HGX B300 systems and others), InfiniBand and Ethernet switches, adapters, cables and transceivers, and the UFM licences at the right tier, specified together so the fabric matches the GPUs. You can browse DGX systems, networking and NVIDIA software in our shop.
Set up. We install UFM (in an HA pair where it fits), configure partitions and the Slurm or Kubernetes integration, deploy DCGM and the OpenTelemetry collectors, and hand over dashboards and alert rules that have been tested against real faults: we pull a cable and show you the page arrive.
Run it for you. Our managed observability service keeps the monitoring stack itself healthy and highly available, tunes alerts so they are worth waking up for, watches fabric and GPU health around the clock, and sends you a monthly report on utilisation, faults and capacity. We work alongside your team and NVIDIA support, not instead of them.

Questions to ask before you buy

Who will own the fabric day to day, and do they need to run it or just see it?
Will the cluster be shared between teams, departments or customers? If so, how will they be isolated on the network?
Which scheduler do you use, and do you want GPU and fabric usage reported per job and per team?
Where should the monitoring data live, and for how long? Do you already run Prometheus, Grafana or Elastic?
What does an hour of your cluster cost, and how quickly do you need to know when it is being wasted?

NVIDIA partner: engineering, software and people

Workstation is an NVIDIA partner. Beyond supplying the hardware, we provide the engineering, software and staff to plan, build and run AI data centres: site and power surveys, rack and fabric design, InfiniBand and Spectrum-X installation, UFM, DCGM and OpenTelemetry set-up, Kubernetes and Slurm onboarding, and engineers who stay with you through handover and managed support. For NVIDIA's own description of the platform, see the official NVIDIA Unified Fabric Manager (UFM) page (opens in a new tab).

Next steps

Open the interactive dashboard gallery to see the seventeen views we build for a typical cluster, from fabric health to cost per GPU-hour. Then ask our AI Sales Engineer to size a cluster, a fabric and the right UFM tier for your workloads, or browse the Workstation shop for HGX B300 servers and DGX BasePOD configurations. We are happy to start with an observability review of a cluster you already own.

Figures in this post are illustrative examples, not measurements from a specific customer.

Acknowledgement of rights: NVIDIA, the NVIDIA logo, DGX, HGX, Quantum, Spectrum, Spectrum-X, BlueField, ConnectX, NVLink, DOCA, DCGM and Unified Fabric Manager (UFM) are trademarks and/or registered trademarks of NVIDIA Corporation in the United States and other countries. NVIDIA products and technologies may be covered by patents and other intellectual property rights owned by NVIDIA Corporation, and all such rights are acknowledged and reserved to their owners. Product information is summarised in our own words from NVIDIA's public documentation; NVIDIA's website and documentation are the authoritative source. Other company and product names, including Prometheus, Grafana, Elasticsearch, Kibana, Logstash, OpenTelemetry, Kubernetes and Slurm, may be trademarks of their respective owners. Workstation's own dashboards, configurations and text are © Workstation.

Technote · for platform, network and HPC engineers

NVIDIA Observability with OpenTelemetry: UFM SDK Plugins, DCGM, DOCA and NetQ into Mimir, Grafana and the Elastic Stack

A practitioner's guide to NVIDIA's observability tooling for AI clusters and how we join it up with OpenTelemetry. It covers UFM and its open SDK plugins, the UFM REST API and telemetry endpoints, DCGM and dcgm-exporter for GPUs (including the metric renames in dcgm-exporter 4.9.0), DOCA Telemetry Service on BlueField, NetQ for Spectrum-X Ethernet, and Workstation's reference design: an OpenTelemetry Collector agent on every node, a gateway pair, and two backends (Prometheus-compatible metrics in Grafana Mimir with Grafana and Tempo, and the Elastic stack). Working configuration, alert rules, a labelling scheme, cardinality sums and the high-availability design for the monitoring itself are included. The business case is in the companion blog post.

Telemetry from GPUs, InfiniBand and Ethernet switches, BlueField DPUs and AI services flows through OpenTelemetry Collectors to Prometheus, Mimir and Grafana, and to Elasticsearch and Kibana.

Summary. Collect each signal once, as close to its source as is sensible, label it with one fleet-wide scheme, and fan it out to the backends people already use. GPU counters come from DCGM, InfiniBand fabric counters and events from UFM, Ethernet fabric data from NetQ or the switches, DPU counters from DOCA Telemetry Service, and request traces from the AI services through the OpenTelemetry SDK. OpenTelemetry Collectors do the labelling, filtering, sampling and routing. Metrics and traces go to Mimir and Tempo, logs and events go to Elasticsearch, and the whole pipeline is itself monitored and highly available.

This article follows the stance of our earlier Big Prometheus and MoSCoW technote: OpenTelemetry is the telemetry contract, Prometheus semantics and PromQL are the metrics language, and a horizontally scalable store (here Grafana Mimir) is used once a single Prometheus no longer answers fleet-wide questions. The one thing we add for AI clusters is a second backend for logs, because the Elastic stack is so often already in place for security and operations teams. Logs go to one primary store, as that article recommends; we do not write the same logs twice.

1. The signals that matter in an AI cluster

An AI cluster fails in layers. A useful monitoring design names each layer, the NVIDIA tool that knows about it, and how the data leaves that tool.

Layer	Typical faults	NVIDIA source	How we collect it
GPU and NVLink	XID errors, ECC and row-remap events, thermal or power throttling, NVLink errors, idle GPUs in running jobs	DCGM (host engine) and dcgm-exporter	Prometheus scrape on the node by the Collector agent
InfiniBand compute and storage fabric	Link flaps, symbol and integrity errors, links trained at the wrong speed or width, congestion, hot or failing transceivers, subnet manager changes	UFM Telemetry and UFM Enterprise (events, alarms, topology, REST API), SDK plugins	Prometheus scrape of UFM telemetry; UFM events by syslog or Fluentd; REST polling for topology
Ethernet fabric (Spectrum-X, storage and front end)	Drops, PFC storms, ECN marking, RoCE congestion, BGP or EVPN changes	NetQ; switch streaming telemetry	NetQ API or switch gNMI or OTLP streaming into the gateway
BlueField DPU	Port and offload counters, DPU health	DOCA Telemetry Service	Scrape of its Prometheus-format endpoint
Host and storage clients	Memory pressure, NVMe wear, file-system latency, NIC errors	(standard Linux tooling)	node-exporter, filelog and journald receivers
Scheduler and orchestration	Jobs stuck, nodes drained, GPU requests with no utilisation	(Slurm, Kubernetes), UFM Slurm integration	Job-to-GPU labels from dcgm-exporter; scheduler exporters; Kubernetes metadata
AI services	Slow time to first token, queueing, errors, token throughput drops	(model servers such as NIM, Triton or vLLM)	OpenTelemetry SDK traces and metrics over OTLP

2. NVIDIA UFM: tiers, telemetry endpoints and REST API

Unified Fabric Manager (see the official NVIDIA UFM page) runs and observes InfiniBand fabrics. It comes in three tiers that build on each other:

UFM Telemetry samples performance and error counters from every port in the fabric, together with cable and transceiver data, and exposes or streams them. It is a visibility layer only.
UFM Enterprise adds fabric management: discovery and topology, subnet manager control, partition key (PKey) management, events and alarms, health and topology reports, the REST API, integrations with schedulers and cluster managers, and an active/standby high-availability pair.
UFM Cyber-AI applies analytics and machine learning to the Enterprise telemetry history to predict failing links and cables and to flag performance or usage anomalies.

Telemetry endpoints

UFM's telemetry service serves its counters over HTTP, by default on port 9001. There is a CSV view (for example /csv/metrics, plus counter-set views such as /csv/xcset/ib_basic_debug) and, when Prometheus label generation is enabled, a Prometheus endpoint whose path the SDK documentation gives as labels/enterprise. Counter-set views accept sharding parameters (num_shards, shard, sharding_field=port_guid), which helps when a single request for a large fabric is too big to answer within a scrape interval.

Counter names follow the InfiniBand performance-management attributes. The ones we have confirmed in NVIDIA's own SDK material, and which we build on below, are:

Counter	What it tells you
`PortXmitDataExtended`, `PortRcvDataExtended`	Data sent and received on the port; rate it for bandwidth (NVIDIA's own Grafana dashboard multiplies the rate by 8 to show bits per second)
`LinkDownedCounterExtended`	Times the link went down: the link-flap signal
`LinkErrorRecoveryCounterExtended`	Times the link retrained to recover from errors: often the first sign of a marginal cable
`LocalLinkIntegrityErrorsExtended`	Local physical errors exceeding the integrity threshold
`ExcessiveBufferOverrunErrorsExtended`	Consecutive buffer overruns on the port
`Normalized_CBW`	A congestion bandwidth figure UFM exposes; NVIDIA's dashboard plots its maximum by topology level
`dev_temperature`	Device temperature

The exact set and label names depend on the counter set you enable and the UFM version, so always list what your endpoint actually serves before writing rules: curl -s http://<ufm-host>:9001/csv/metrics | head.

REST API

UFM Enterprise's REST API lives under /ufmRest on the UFM server, with sibling prefixes for other authentication methods: /ufmRestV2 (session), /ufmRestV3 (access token) and /ufmRestKrb (Kerberos). Tokens are created with a POST to /app/tokens. For monitoring we use three read-only resources, all of which appear in NVIDIA's SDK examples:

# Current alarms and recent events (use a read-only monitoring account)
curl -sk -u monitor:"$UFM_PASSWORD" https://ufm-vip.obs.internal/ufmRest/app/alarms
curl -sk -u monitor:"$UFM_PASSWORD" https://ufm-vip.obs.internal/ufmRest/app/events

# Every link in the fabric, with both ends: the basis of our node-to-switch-port map
curl -sk -u monitor:"$UFM_PASSWORD" https://ufm-vip.obs.internal/ufmRest/resources/links

# UFM's own health in Prometheus format (scrape every few minutes, not seconds)
curl -sk -u monitor:"$UFM_PASSWORD" https://ufm-vip.obs.internal/ufmRest/system_monitoring/metrics

We wrap the links call in a small exporter that publishes an info metric, ws_ib_link_info{node, hca_port, switch, port}, every five minutes. That metric is what lets a query walk from a Slurm job to its GPUs to the exact switch ports its traffic crosses. It is our convention, not an NVIDIA metric.

High availability

UFM Enterprise runs as an active/standby pair managed by its HA tooling (ufm_ha_cluster), with configuration replicated between the two servers and a virtual IP that moves on failover. Point every scrape, REST client and syslog destination at the virtual IP, never at a node. Where a plugin is used in HA, NVIDIA's instructions are to load its container image on the standby node as well, so it is ready after failover.

3. The UFM SDK and its plugins

NVIDIA publishes an open SDK for UFM at github.com/Mellanox/ufm_sdk_3.0. Most of it is a set of plugins: containers that UFM manages through its plugin manager (from the web UI, or with /opt/ufm/scripts/manage_ufm_plugins.sh add -p <name> on a standalone or HA install). Here is what each one does and how it fits our design.

Plugin	What it does	Where it fits
`grafana_infiniband_telemetry_plugin`	Serves a second Prometheus endpoint (port `8982`, path `/labels/enterprise`) with human-readable labels such as device names instead of GUID-based labels, and ships a sample Grafana dashboard.	Our preferred scrape target for fabric counters: readable labels mean fewer relabelling rules.
`grafana_telemetry_plugin`	Instructions and a dashboard for scraping UFM Telemetry's own Prometheus endpoint (port 9001) directly.	Use when you cannot run the plugin above; expect GUID-based labels.
`grafana_ufm_monitoring_plugin`	A dashboard for UFM's own system health, scraped from `/ufmRest/system_monitoring/metrics` over HTTPS with basic authentication.	Monitors the monitor: CPU, memory and API health of the UFM servers.
`ufm_events_grafana_dashboard_plugin`	Parses UFM's `event.log` with Fluentd, pushes it to a bundled Loki and shows it in a bundled Grafana.	Good for a quick start; in production we send events to the central log store instead.
`fluentd_telemetry_plugin` (TFS)	Polls one or more UFM telemetry endpoints at its own interval per endpoint and streams the counters to a Fluentd destination (forward protocol on 24224 by default, or HTTP). Attributes can be enabled, disabled or renamed through its REST API, and it publishes its own streaming statistics in Prometheus format.	The route for fabric counters into Elasticsearch when a customer has standardised on Elastic for metrics as well as logs.
`fluentd_topology_plugin`	Extracts topology through the UFM API and streams it over the Fluentd protocol.	Topology snapshots in the log store, so you can see what the fabric looked like at the time of an incident.
`ufm_syslog_streaming_plugin` (EFS)	Extracts UFM events from its syslog and streams them to Fluentd (Fluent Bit compatible), and can also copy UFM syslog to a remote syslog server.	Our main route for UFM events and alarms into Elasticsearch.
`zabbix_telemetry_plugin`	Instructions for streaming UFM events into Zabbix through the Zabbix agent and log items.	For operations teams that already run Zabbix as their event console.
`SLURM-Integration`	Slurm prolog and epilog hooks that call UFM for every job: static or dynamic (per-job, derived from the job ID) partition keys, with UFM managing SHARP reservations automatically from PKey membership. Supports token, basic or Kerberos authentication. UFM must run in management mode, not monitoring-only.	Per-job network isolation and a record of which job used which ports.
`pdr_deterministic_plugin`	Problem Detection and Resolution: uses UFM dynamic telemetry to compute bit error rates, temperatures and link-down deltas, isolates ports that cross thresholds, records them, and de-isolates them after a clean test period. Has a dry-run mode and an exclusion list.	Automatic isolation of bad links so jobs route around them; see section 4.
`UFM_NDT_Plugin`	Network Design Tool comparison: upload the intended topology, compare it with what UFM discovered, once or on a schedule, and read the reports through the REST API.	Proves the as-built cabling matches the design at handover and after maintenance.
`gnmi_nvos_events_plugin`	Subscribes to "on change" system events from NVOS-managed switches over gNMI and turns them into UFM external events.	Brings switch-side events (for example a transceiver reset) into the same UFM event stream.
`snmp_receiver_plugin`	Listens for SNMP traps from managed switches and redirects them into UFM as events.	Same idea for switches that only send traps.
`grpc_streamer_plugin`	A gRPC server that returns UFM REST data (events, alarms, links and others) once or as a periodic stream, optionally only the changes. NVIDIA marks it as under development.	Not in our production design yet; we watch it as a lower-latency alternative to REST polling.
`ufm_log_analyzer_plugin`	Analyses a UFM sysdump or logs offline and produces findings and a PDF report. NVIDIA marks it as under development and for internal use.	Useful for support cases; not part of live monitoring.
`bright_plugin`	Connects UFM to Bright Cluster Manager (now NVIDIA Base Command Manager) and pulls node status and job assignments into UFM.	Puts workload context next to network events for root-cause work on BCM-managed clusters.

Most plugins ship as images on Docker Hub (mellanox/ufm-plugin-*); PDR is downloaded from the NVIDIA Licensing Portal. For air-gapped sites, the SDK documents the usual docker save and docker load route, and we include the images in the offline bundle we stage before shipping.

4. Problem Detection and Resolution in practice

PDR is the plugin that changes operations most, so it deserves its own section. Its configuration file exposes the thresholds directly. The defaults in the SDK at the time of writing are:

Setting	Default	Meaning
`INTERVAL`	300 s	How often telemetry counters are requested
`MAX_NUM_ISOLATE`	10	Upper bound on isolated ports (the larger of this and 0.5% of fabric size)
`TMAX` / `D_TMAX`	70 °C / 10	Maximum temperature and maximum temperature rise
`MAX_PDR`	1e-12	Maximum packet drop rate
`CONFIGURED_BER_CHECK`, `CONFIGURED_TEMP_CHECK`, `LINK_DOWN_ISOLATION`	True	Which checks may trigger isolation
`SWITCH_TO_HOST_ISOLATION`	True	Whether access links to hosts may be isolated
`DRY_RUN`	False	Log decisions instead of acting
`DEISOLATE_CONSIDER_TIME`	5 min	How long a port must look healthy before it is returned to service
`MAX_PORT_RESET_NUM`	2 per week	Port resets allowed before a reset window clears

Its bit-error-rate check watches symbol errors over short and long windows (about 12 and 125 minutes), which is why it catches cables that are degrading rather than only those that have failed. Our rollout is always the same:

Enable with DRY_RUN=True for two to four weeks and ship its log to Elasticsearch. Every "would isolate" line becomes a ticket we review against UFM counters.
Agree the exclusion list (ports PDR must never touch, such as single-homed storage links) through the plugin's REST API.
Switch to live isolation, with SWITCH_TO_HOST_ISOLATION decided per site: isolating a host link removes that node from jobs, which is sometimes what you want and sometimes not.
Alert on every isolation and de-isolation, and create a maintenance ticket for the physical fix.

5. GPU telemetry: DCGM and dcgm-exporter

NVIDIA Data Center GPU Manager (DCGM) is the GPU health and telemetry service; dcgm-exporter turns its fields into Prometheus metrics, by default on port 9400. On Kubernetes it is normally deployed by the NVIDIA GPU Operator; on Slurm clusters we run it as a systemd service or container next to the DCGM host engine.

dcgm-exporter 4.9.0 renamed the default metrics

Release 4.9.0 (November 2026) renames the default and profiling fields to their DCGM 4.7 names, and eleven of them change unit: clocks to hertz, energy to joules, utilisation from a percentage to a 0 to 1 ratio, memory from MiB to bytes, and NVLink throughput to bytes per second. Dashboards and alerts written for 4.8.x will silently go blank, or worse, compare a ratio with a percentage. The defaults we use most:

4.8.x name	4.9.0 name	Unit change
`DCGM_FI_DEV_GPU_UTIL`	`DCGM_FI_DEV_GPU_UTIL_RATIO`	% to 0–1
`DCGM_FI_DEV_FB_USED`	`DCGM_FI_DEV_FB_USED_BYTES`	MiB to bytes
`DCGM_FI_DEV_GPU_TEMP`	`DCGM_FI_DEV_GPU_TEMP_CELSIUS`	none
`DCGM_FI_DEV_POWER_USAGE`	`DCGM_FI_DEV_BOARD_POWER_WATTS`	none
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	`DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL`	mJ to J
`DCGM_FI_DEV_XID_ERRORS`	`DCGM_FI_DEV_XID_ERROR`	none (last XID value)
`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`	`DCGM_FI_DEV_NVLINK_THROUGHPUT_BYTES_PER_SECOND`	MB/s to B/s
`DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS`	`DCGM_FI_DEV_ROW_REMAP_UNCORRECTABLE_TOTAL`	none
`DCGM_FI_DEV_ROW_REMAP_FAILURE`	`DCGM_FI_DEV_ROW_REMAP_FAILED`	none
`DCGM_FI_DEV_PCIE_REPLAY_COUNTER`	`DCGM_FI_DEV_PCIE_REPLAY_TOTAL`	none
`DCGM_FI_PROF_GR_ENGINE_ACTIVE`	`DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO`	none (ratio)
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	`DCGM_FI_PROF_TENSOR_UTIL_RATIO`	none (ratio)
`DCGM_FI_PROF_DRAM_ACTIVE`	`DCGM_FI_PROF_DRAM_UTIL_RATIO`	none (ratio)

The driver-version label also changes, from DCGM_FI_DRIVER_VERSION to DCGM_FI_SYSTEM_DRIVER_VERSION. Read NVIDIA's upgrade checklist before upgrading. During a mixed-version rollout we bridge the two with recording rules, so dashboards only ever query the ws: names:

groups:
  - name: dcgm-bridge
    rules:
      - record: ws:gpu_util:ratio
        expr: DCGM_FI_DEV_GPU_UTIL_RATIO or (DCGM_FI_DEV_GPU_UTIL / 100)
      - record: ws:gpu_fb_used:bytes
        expr: DCGM_FI_DEV_FB_USED_BYTES or (DCGM_FI_DEV_FB_USED * 1048576)
      - record: ws:gpu_power:watts
        expr: DCGM_FI_DEV_BOARD_POWER_WATTS or DCGM_FI_DEV_POWER_USAGE

Every example in the rest of this article uses the 4.9.0 names.

Which GPU metric means "busy"?

DCGM_FI_DEV_GPU_UTIL_RATIO only says that a kernel was running during the sample; a GPU waiting on the network inside a collective can still read close to 1. For efficiency reporting we use the profiling fields: DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO and DCGM_FI_PROF_TENSOR_UTIL_RATIO. They need DCGM's profiling module and can conflict with other profilers such as Nsight Compute on the same GPU, so we pause profiling metric collection when engineers profile by hand.

Job and pod labels

On Kubernetes, dcgm-exporter adds namespace, pod and container labels for the workload using each GPU. On Slurm, its HPC job mapping reads a directory of per-GPU files that a Slurm prolog writes and the epilog clears, and adds an hpc_job label. Version 4.9.0 hardens this (it rejects symbolic links in the mapping directory and limits file and job-ID sizes), and renames any pod label called hpc_job so the two do not clash.

NVLink and NVSwitch

Inside an HGX or DGX B300 system, GPUs talk over NVLink through NVSwitch. The default set gives total NVLink throughput per GPU; the NVLink error counters are optional and differ by GPU generation, so we enable the right ones in the counters CSV: on Blackwell systems such as B300, the recovery counters (DCGM_FI_DEV_NVLINK_RECOVERY_EVENT_TOTAL, DCGM_FI_DEV_NVLINK_RECOVERY_SUCCESSFUL_TOTAL and DCGM_FI_DEV_NVLINK_RECOVERY_FAILED_TOTAL); on Hopper, the data-link counters (DCGM_FI_DEV_NVLINK_CRC_ERROR_TOTAL, DCGM_FI_DEV_NVLINK_REPLAY_TOTAL, DCGM_FI_DEV_NVLINK_RECOVERY_TOTAL). The older …_CRC_FLIT_ERROR_TOTAL and …_REPLAY_ERROR_TOTAL fields apply only to pre-Hopper GPUs. Rack-scale NVL72 systems add an NVLink switch fabric between trays with its own management software, and we treat it as a separate fabric in the labelling scheme.

6. BlueField DPUs and Spectrum-X Ethernet

DOCA Telemetry Service (DTS) runs as a container on BlueField DPUs (and can run on hosts with ConnectX adapters). It gathers counters from the DPU and its ports through a set of providers and exposes them, among other options, on a Prometheus-format endpoint. We scrape that endpoint from the fabric collector with a file-based target list of DPU management addresses, and label each DPU with the host it serves.

NetQ is NVIDIA's monitoring and validation tool for Ethernet fabrics built on Cumulus Linux and SONiC, including Spectrum-X. It collects state and counters from agents on the switches, keeps history so you can ask what changed and when, and validates the fabric against its intended configuration. For RoCE-based AI fabrics, the counters that matter most are drops, PFC pause frames, ECN-marked packets and buffer occupancy. There are three practical ways to bring this into the central stack, and we choose per site:

keep NetQ as the network team's tool and pull summaries and events from its API into the log store;
stream directly from the switches (gNMI, or OTLP on recent Cumulus Linux releases; check your version's documentation) into the gateway collectors;
fall back to SNMP and syslog for older switches.

Counter names differ between these paths, so we normalise them into our own recording-rule names (ws:eth_pfc_pause_rx:rate5m, ws:eth_ecn_marked:rate5m, ws:eth_drops:rate5m) and build dashboards only on those.

7. Workstation's OpenTelemetry reference design

Reference architecture: GPU nodes run dcgm-exporter, node-exporter and an OpenTelemetry Collector agent; UFM, NetQ and DOCA endpoints are scraped by a fabric collector; a pair of gateway collectors adds labels, filters and samples and exports to Mimir, Tempo, Grafana and Alertmanager and to Logstash, Elasticsearch and Kibana.

The design has three tiers of Collector:

An agent on every GPU, storage and login node, running as a systemd service or a Kubernetes DaemonSet. It scrapes local exporters over loopback, tails local logs, receives OTLP from local services, adds host and Kubernetes metadata, and forwards everything to the gateways. It keeps a small persistent queue so a gateway restart loses nothing.
A gateway pair (or more), behind a virtual IP or DNS name. Gateways apply fleet-wide policy: labels, attribute clean-up, filtering, trace tail sampling, and routing to backends. Credentials for the backends live only here.
A fabric pipeline on the gateway hosts that scrapes UFM, DOCA and switch endpoints and receives syslog. Both gateways run it; the duplicate samples are removed in Mimir by its HA tracker (section 16).

Why not have every node write straight to the backends? Because the gateway is where policy changes happen once, where secrets stay off hundreds of nodes, and where a backend migration (say, adding Elasticsearch for a security team) is a configuration change on two servers rather than a fleet rollout.

8. Collector agent configuration

This is the agent we deploy on a Slurm GPU node, trimmed only of site-specific certificates. It needs the otelcol-contrib distribution, or a custom build that includes the same components.

# /etc/otelcol-contrib/config.yaml  (agent, one per node)
extensions:
  health_check:
    endpoint: 127.0.0.1:13133
  file_storage:
    directory: /var/lib/otelcol/queue
    create_directory: true

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

  prometheus:
    config:
      scrape_configs:
        - job_name: dcgm
          scrape_interval: 15s
          static_configs:
            - targets: ["127.0.0.1:9400"]
        - job_name: node
          scrape_interval: 15s
          static_configs:
            - targets: ["127.0.0.1:9100"]
          metric_relabel_configs:
            # keep what the dashboards use; node-exporter's full output is large
            - source_labels: [__name__]
              regex: "node_(cpu_seconds_total|load1|load5|memory_.+|pressure_.+|filesystem_(avail|size)_bytes|disk_(read|written)_bytes_total|network_(receive|transmit)_(bytes|errs|drop)_total|infiniband_.+|hwmon_temp_celsius|nvme_.+|textfile_.+)"
              action: keep

  filelog:
    include:
      - /var/log/syslog
      - /var/log/slurm/slurmd.log
      - /var/log/slurm/ufm_slurm.log
    include_file_path: true
    start_at: end
    storage: file_storage

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  resource/fleet:
    attributes:
      - key: site
        value: ${env:WS_SITE}
        action: upsert
      - key: cluster
        value: ${env:WS_CLUSTER}
        action: upsert
      - key: rack
        value: ${env:WS_RACK}
        action: upsert
      - key: node_role
        value: ${env:WS_NODE_ROLE}
        action: upsert
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp/gateway:
    endpoint: otel-gateway.obs.internal:4317
    tls:
      ca_file: /etc/otelcol-contrib/ca.pem
    sending_queue:
      enabled: true
      storage: file_storage
      queue_size: 5000
    retry_on_failure:
      enabled: true
      max_elapsed_time: 30m
  # traces are routed by trace ID so every span of a trace reaches the same
  # gateway, which tail sampling needs
  loadbalancing/traces:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          ca_file: /etc/otelcol-contrib/ca.pem
    resolver:
      dns:
        hostname: otel-gateway-pool.obs.internal
        port: "4317"

service:
  extensions: [health_check, file_storage]
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, resourcedetection, resource/fleet, batch]
      exporters: [otlp/gateway]
    logs:
      receivers: [otlp, filelog]
      processors: [memory_limiter, resourcedetection, resource/fleet, batch]
      exporters: [otlp/gateway]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, resource/fleet, batch]
      exporters: [loadbalancing/traces]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

On Kubernetes nodes we add the k8sattributes processor to the agent (not the gateway, because by the time data reaches a gateway the original pod IP is gone) and filter it to the local node:

processors:
  k8sattributes:
    auth_type: serviceAccount
    filter:
      node_from_env_var: K8S_NODE_NAME
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.deployment.name
        - k8s.node.name
      labels:
        - tag_name: tenant
          key: workstation.co.uk/tenant
          from: namespace

9. Gateway configuration and the two backends

The gateway receives OTLP from agents, runs the fabric scrapes and syslog, and exports to both backends. There are two metrics pipelines: one for data forwarded from agents (unique, so no replica label) and one for the fabric scrapes that both gateways run (labelled with the gateway's replica name so Mimir keeps one copy).

# /etc/otelcol-contrib/config.yaml  (gateway, two or more)
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage:
    directory: /var/lib/otelcol/queue
    create_directory: true
  basicauth/ufm:
    client_auth:
      username: ${env:UFM_MONITOR_USER}
      password: ${env:UFM_MONITOR_PASSWORD}
  basicauth/es:
    client_auth:
      username: ${env:ES_USER}
      password: ${env:ES_PASSWORD}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol-contrib/tls/gateway.crt
          key_file: /etc/otelcol-contrib/tls/gateway.key

  prometheus/fabric:
    config:
      scrape_configs:
        - job_name: ufm-telemetry
          scrape_interval: 60s
          scrape_timeout: 50s
          metrics_path: /labels/enterprise
          static_configs:
            - targets: ["ufm-vip.obs.internal:8982"]
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "PortXmitDataExtended|PortRcvDataExtended|LinkDownedCounterExtended|LinkErrorRecoveryCounterExtended|LocalLinkIntegrityErrorsExtended|ExcessiveBufferOverrunErrorsExtended|Normalized_CBW|dev_temperature"
              action: keep
        - job_name: ufm-system
          scrape_interval: 4m
          scrape_timeout: 60s
          scheme: https
          metrics_path: /ufmRest/system_monitoring/metrics
          basic_auth:
            username: ${env:UFM_MONITOR_USER}
            password: ${env:UFM_MONITOR_PASSWORD}
          tls_config:
            ca_file: /etc/otelcol-contrib/ufm-ca.pem
          static_configs:
            - targets: ["ufm-vip.obs.internal:443"]
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "rest_api_response_time_histogram.*"
              action: drop
        - job_name: doca-telemetry
          scrape_interval: 30s
          file_sd_configs:
            - files: ["/etc/otelcol-contrib/targets/dpu-*.yaml"]

  syslog/fabric:
    tcp:
      listen_address: 0.0.0.0:5514
    protocol: rfc5424

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 20
  resource/fabric:
    attributes:
      - key: site
        value: ${env:WS_SITE}
        action: upsert
      - key: cluster
        value: ${env:WS_CLUSTER}
        action: upsert
      - key: fabric
        value: ib-compute
        action: upsert
  transform/hygiene:
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "pod_uid")
          - delete_key(attributes, "container_id")
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "http.request.header.authorization")
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  prometheusremotewrite/mimir:
    endpoint: https://mimir.obs.internal/api/v1/push
    headers:
      X-Scope-OrgID: ${env:WS_TENANT}
    resource_to_telemetry_conversion:
      enabled: true
    remote_write_queue:
      enabled: true
      queue_size: 20000
      num_consumers: 8
    retry_on_failure:
      enabled: true
      max_elapsed_time: 30m
  prometheusremotewrite/mimir-ha:
    endpoint: https://mimir.obs.internal/api/v1/push
    headers:
      X-Scope-OrgID: ${env:WS_TENANT}
    external_labels:
      replica: ${env:WS_GATEWAY_NAME}
    resource_to_telemetry_conversion:
      enabled: true
  otlp/tempo:
    endpoint: tempo-distributor.obs.internal:4317
    tls:
      ca_file: /etc/otelcol-contrib/ca.pem
    sending_queue:
      enabled: true
      storage: file_storage
  elasticsearch/logs:
    endpoints:
      - https://es-hot-1.obs.internal:9200
      - https://es-hot-2.obs.internal:9200
    logs_index: logs-ai.cluster-prod
    # ECS field names, so existing Kibana views and detection rules keep working
    mapping:
      allowed_modes: [ecs]
    auth:
      authenticator: basicauth/es
    tls:
      ca_file: /etc/otelcol-contrib/es-ca.pem

service:
  extensions: [health_check, file_storage, basicauth/ufm, basicauth/es]
  pipelines:
    metrics/forwarded:
      receivers: [otlp]
      processors: [memory_limiter, transform/hygiene, batch]
      exporters: [prometheusremotewrite/mimir]
    metrics/fabric:
      receivers: [prometheus/fabric]
      processors: [memory_limiter, resource/fabric, batch]
      exporters: [prometheusremotewrite/mimir-ha]
    logs:
      receivers: [otlp, syslog/fabric]
      processors: [memory_limiter, transform/hygiene, batch]
      exporters: [elasticsearch/logs]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/tempo]
  telemetry:
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

Notes on choices:

Scrape interval for fabric counters is 60 seconds. Faster rarely helps: the UFM sample itself, the alert window and human response dominate detection time (see the data-flow figure below), and per-port series are where the volume is.
The keep list on ufm-telemetry is the single most effective cost control in the whole design. Start narrow; add a counter when a dashboard or alert needs it.
resource_to_telemetry_conversion turns resource attributes into labels. That is what puts site, cluster, rack and fabric on every series, and it is also how cardinality accidents happen, which is why the agents set only the attributes we want and the hygiene processor removes known offenders.
Optional Loki. Where a customer prefers Grafana for logs, add an otlphttp/loki exporter pointing at Loki's native OTLP endpoint and send logs there instead of Elasticsearch, not as well.
Traces to Elastic. Customers standardised on Elastic Observability can send traces to Elastic's OTLP intake instead of Tempo; the rest of the design is unchanged.

10. Prometheus scrape configuration (if you run Prometheus)

Some customers keep a Prometheus HA pair per site, either because they already have one or to keep local alerting working if the link to the central store fails. The same targets, expressed as plain Prometheus configuration, look like this. The replica external label is the only thing that differs between the two servers.

# prometheus.yml (replica "a"; replica "b" is identical except for the label)
global:
  scrape_interval: 15s
  evaluation_interval: 30s
  external_labels:
    site: lon2
    cluster: ai-prod-1
    replica: a

rule_files:
  - /etc/prometheus/rules/*.yaml

scrape_configs:
  - job_name: dcgm
    file_sd_configs:
      - files: ["/etc/prometheus/targets/gpu-nodes.yaml"]
    relabel_configs:
      - source_labels: [__address__]
        regex: "([^:]+):.*"
        target_label: node
        replacement: "$1"
        action: replace

  - job_name: ufm-telemetry
    scrape_interval: 60s
    scrape_timeout: 50s
    metrics_path: /labels/enterprise
    static_configs:
      - targets: ["ufm-vip.obs.internal:8982"]
        labels:
          fabric: ib-compute
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "PortXmitDataExtended|PortRcvDataExtended|LinkDownedCounterExtended|LinkErrorRecoveryCounterExtended|LocalLinkIntegrityErrorsExtended|ExcessiveBufferOverrunErrorsExtended|Normalized_CBW|dev_temperature"
        action: keep

  - job_name: otel-collectors
    file_sd_configs:
      - files: ["/etc/prometheus/targets/otel-*.yaml"]

remote_write:
  - url: https://mimir.obs.internal/api/v1/push
    headers:
      X-Scope-OrgID: ai-prod
    queue_config:
      max_samples_per_send: 5000

The target files carry the rack and role labels, so they come from the same source of truth as the agents' environment variables (our inventory), never typed by hand:

# /etc/prometheus/targets/gpu-nodes.yaml
- targets: ["gpu-r07-01:9400", "gpu-r07-02:9400", "gpu-r07-03:9400", "gpu-r07-04:9400"]
  labels:
    rack: r07
    node_role: gpu

11. Fleet-wide labelling scheme

Data flow: a GPU counter read by DCGM, a network adapter counter read by node-exporter or DOCA, and a switch port counter read by UFM Telemetry are collected, labelled with site, cluster, fabric, rack, node and rail, stored in Mimir or Elasticsearch, and used in Grafana, Kibana and alert rules.

Labels are the joins of observability. If GPU metrics say host="gpu-r07-03.lon2" and fabric metrics say node="GPU-R07-03", no query will ever connect them. We fix one scheme for every customer and every source:

Label	Example	On	Rule
`site`	lon2	everything	Data centre or hall; set by the collector
`cluster`	ai-prod-1	everything	Also Mimir's HA cluster label
`env`	prod	everything	prod, staging, lab
`fabric`	ib-compute	network metrics	ib-compute, ib-storage, eth-frontend, eth-storage, nvlink
`rack`	r07	nodes and switches	From inventory, never derived from hostnames at query time
`node`	gpu-r07-03	node metrics	Short hostname, lower case; we relabel DCGM's `Hostname` to match
`rail`	3	GPU and NIC metrics	Rail number in rail-optimised fabrics; equals the GPU index on most HGX designs
`gpu`, `UUID`	3, GPU-…	GPU metrics only	From dcgm-exporter; the UUID follows the physical GPU through RMAs
`switch`, `port`	leaf-r07-a, 17	fabric metrics	Normalised from UFM's device and port labels in relabelling
`tenant`	genomics	workload metrics	From Kubernetes namespace labels or Slurm account
`hpc_job`	4471823	GPU metrics only	Never on fabric or node metrics (see cardinality)

With that in place, the job-to-fabric join is a pair of PromQL joins through our two info metrics, ws_job_node_info{hpc_job, node} (from the scheduler exporter) and ws_ib_link_info{node, hca_port, switch, port} (from the UFM links exporter):

# link-error recoveries in the last hour on every switch port used by job 4471823
sum by (switch, port) (
  increase(LinkErrorRecoveryCounterExtended[1h])
    * on (switch, port) group_left (node)
      (ws_ib_link_info * on (node) group_left () ws_job_node_info{hpc_job="4471823"})
)

12. Alert rules

These rules run in the Mimir ruler (or Prometheus) and route through Alertmanager. Severity page wakes someone up; ticket goes to the queue. Thresholds are starting points we tune per site during the first month.

# rules/ai-cluster.yaml
groups:
  - name: gpu
    rules:
      - alert: GPUXidErrorReported
        expr: changes(DCGM_FI_DEV_XID_ERROR[10m]) > 0 and DCGM_FI_DEV_XID_ERROR != 0
        labels:
          severity: page
        annotations:
          summary: "XID error reported on {{ $labels.node }} GPU {{ $labels.gpu }}"
          runbook: "https://runbooks.obs.internal/gpu/xid"
      - alert: GPURowRemapFailed
        expr: DCGM_FI_DEV_ROW_REMAP_FAILED > 0
        labels:
          severity: page
        annotations:
          summary: "Row remapping failed on {{ $labels.node }} GPU {{ $labels.gpu }}: drain and RMA"
      - alert: GPUUncorrectableRowRemaps
        expr: increase(DCGM_FI_DEV_ROW_REMAP_UNCORRECTABLE_TOTAL[1h]) > 0
        labels:
          severity: ticket
        annotations:
          summary: "New uncorrectable-error row remaps on {{ $labels.node }} GPU {{ $labels.gpu }}"
      - alert: GPUTemperatureHigh
        # set per platform: air-cooled and liquid-cooled systems differ
        expr: max_over_time(DCGM_FI_DEV_GPU_TEMP_CELSIUS[10m]) > 85
        for: 10m
        labels:
          severity: ticket
      - alert: GPUPCIeReplays
        expr: increase(DCGM_FI_DEV_PCIE_REPLAY_TOTAL[15m]) > 100
        labels:
          severity: ticket
      - alert: GPUAllocatedButIdle
        expr: avg by (cluster, hpc_job) (avg_over_time(DCGM_FI_DEV_GPU_UTIL_RATIO{hpc_job!=""}[30m])) < 0.05
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "Job {{ $labels.hpc_job }} has held GPUs for 30 minutes at under 5% utilisation"
      - alert: DCGMExporterDown
        expr: up{job="dcgm"} == 0
        for: 5m
        labels:
          severity: page

  - name: infiniband
    rules:
      - record: ws:ib_port_tx_bps:rate5m
        expr: rate(PortXmitDataExtended[5m]) * 8
      - record: ws:ib_port_rx_bps:rate5m
        expr: rate(PortRcvDataExtended[5m]) * 8
      - alert: IBLinkFlapping
        expr: increase(LinkDownedCounterExtended[15m]) >= 2
        labels:
          severity: page
        annotations:
          summary: "Link flapping on {{ $labels.switch }} port {{ $labels.port }}"
      - alert: IBLinkErrorRecovery
        expr: increase(LinkErrorRecoveryCounterExtended[15m]) > 0
        labels:
          severity: ticket
        annotations:
          summary: "Link retrained after errors on {{ $labels.switch }} port {{ $labels.port }}: check cable and transceiver"
      - alert: IBLinkIntegrityErrors
        expr: increase(LocalLinkIntegrityErrorsExtended[15m]) > 0
        labels:
          severity: ticket
      - alert: IBBufferOverruns
        expr: increase(ExcessiveBufferOverrunErrorsExtended[15m]) > 0
        labels:
          severity: ticket
      - alert: IBDeviceHot
        # PDR's default maximum temperature is also 70 C
        expr: dev_temperature > 70
        for: 10m
        labels:
          severity: ticket
      - alert: UFMTelemetryDown
        expr: up{job="ufm-telemetry"} == 0
        for: 5m
        labels:
          severity: page
      - alert: UFMTelemetryMissing
        expr: absent_over_time(PortXmitDataExtended{fabric="ib-compute"}[10m])
        labels:
          severity: page

  - name: pipeline
    rules:
      - alert: CollectorExportFailing
        expr: sum by (instance, exporter) (rate(otelcol_exporter_send_failed_metric_points_total[5m])) > 0
        for: 10m
        labels:
          severity: page
      - alert: CollectorQueueFilling
        expr: max by (instance, exporter) (otelcol_exporter_queue_size / otelcol_exporter_queue_capacity) > 0.8
        for: 10m
        labels:
          severity: ticket
      - alert: GatewayRedundancyLost
        expr: count(up{job="otel-gateway"} == 1) < 2
        for: 5m
        labels:
          severity: page
      - alert: ElasticsearchClusterRed
        expr: max(elasticsearch_cluster_health_status{color="red"}) == 1
        for: 2m
        labels:
          severity: page
      - alert: Watchdog
        # always firing; an external dead man's switch pages if it stops arriving
        expr: vector(1)
        labels:
          severity: heartbeat

Two rules of thumb. First, every page links a runbook that says what to do in the first ten minutes. Second, alerts that fire on counters with increase() need a window at least four times the scrape interval, or a single missed scrape hides the event; that is why the fabric windows are 15 minutes for 60-second scrapes.

13. Logs and events into the Elastic stack

UFM events, switch syslog, PDR decisions, Slurm and UFM-Slurm logs, and host logs go to Elasticsearch. Node logs arrive through the Collector pipeline above. UFM's own streams arrive through its Fluentd-based plugins, which we terminate on a Fluentd aggregator (or Logstash) next to the gateways.

Fluentd: UFM event and telemetry streams to Elasticsearch

The EFS (syslog streaming) and TFS (telemetry) plugins send over the Fluentd forward protocol. This aggregator receives both, adds the fleet labels and writes to data streams with a disk buffer, so an Elasticsearch outage does not lose events. It uses the fluent-plugin-elasticsearch output.

# /etc/fluent/fluentd.conf  (aggregator next to the gateways)
<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<filter **>
  @type record_transformer
  <record>
    site lon2
    cluster ai-prod-1
    fabric ib-compute
  </record>
</filter>

<match **>
  @type elasticsearch_data_stream
  data_stream_name logs-ufm.events-prod
  hosts https://es-hot-1.obs.internal:9200,https://es-hot-2.obs.internal:9200
  user "#{ENV['ES_USER']}"
  password "#{ENV['ES_PASSWORD']}"
  ca_file /etc/fluent/es-ca.pem
  <buffer>
    @type file
    path /var/lib/fluentd/buffer/es
    flush_interval 10s
    chunk_limit_size 8MB
    total_limit_size 4GB
    retry_max_interval 60s
    retry_forever true
  </buffer>
</match>

In practice we split this with two <match> blocks on the tags the two plugins are configured to send, so events and telemetry land in separate data streams with different retention. Sending per-port telemetry counters to Elasticsearch is only worth it when Elastic is the customer's single metrics store; otherwise counters go to Mimir and Elasticsearch gets events.

Logstash: switch syslog

For sites with an existing Logstash tier, switch and UFM syslog can land there instead of on the Collector's syslog receiver:

# /etc/logstash/conf.d/fabric-syslog.conf
input {
  syslog {
    port => 5514
  }
}

filter {
  if [message] =~ /(?i)link (down|up)|transceiver|module (inserted|removed)/ {
    mutate { add_tag => ["fabric_link_event"] }
  }
  mutate {
    add_field => {
      "site"    => "lon2"
      "cluster" => "ai-prod-1"
    }
  }
}

output {
  elasticsearch {
    hosts => ["https://es-hot-1.obs.internal:9200", "https://es-hot-2.obs.internal:9200"]
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_dataset => "network.switch"
    data_stream_namespace => "prod"
    user => "${ES_USER}"
    password => "${ES_PASSWORD}"
    ssl_certificate_authorities => ["/etc/logstash/es-ca.pem"]
  }
}

Retention with index lifecycle management

PUT _ilm/policy/ai-cluster-logs
{
  "policy": {
    "phases": {
      "hot":    { "actions": { "rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" } } },
      "warm":   { "min_age": "7d",  "actions": { "forcemerge": { "max_num_segments": 1 } } },
      "cold":   { "min_age": "30d", "actions": { "set_priority": { "priority": 0 } } },
      "delete": { "min_age": "90d", "actions": { "delete": {} } }
    }
  }
}

Security-relevant streams (UFM audit, authentication) usually need longer retention than 90 days; give them their own data stream and policy rather than lengthening everything.

Useful Kibana queries (KQL) to save as views:

data_stream.dataset : "network.switch" and tags : "fabric_link_event" and host.name : leaf-r07-*
data_stream.dataset : "ufm.events" and message : (*isolat* or *deisolat*)
data_stream.dataset : "ufm.events" and log.level : ("critical" or "error") and not message : *test*

14. Tracing AI inference and training services

Metrics tell you that time to first token went up; traces tell you where the time went. For an inference request the useful spans are the request, the tokeniser, the call to the model server, and the model server's own queueing, prefill and decode. Recent model servers (vLLM, Triton and NVIDIA NIM among them) can emit OpenTelemetry traces themselves when given an OTLP endpoint; check the flags for your version. The application side looks like this, using the OpenTelemetry GenAI semantic-convention attribute names:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider(resource=Resource.create({
    "service.name": "chat-gateway",
    "deployment.environment.name": "prod",
    "cluster": "ai-prod-1",
}))
# the local Collector agent; it adds node and rack labels
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://127.0.0.1:4317", insecure=True)))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("ws.inference")


def handle_chat(req, tokenizer, model_client):
    with tracer.start_as_current_span("chat.completions", kind=trace.SpanKind.SERVER) as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.request.model", req.model)

        with tracer.start_as_current_span("tokenize") as s:
            ids = tokenizer.encode(req.prompt)
            s.set_attribute("gen_ai.usage.input_tokens", len(ids))

        # the client propagates the W3C traceparent header, so the model
        # server's own spans (queue, prefill, decode) join this trace
        with tracer.start_as_current_span("model_server.generate", kind=trace.SpanKind.CLIENT) as s:
            out = model_client.generate(ids, max_tokens=req.max_tokens)
            s.set_attribute("gen_ai.usage.output_tokens", out.output_tokens)
            s.set_attribute("server.address", out.server)
            if out.gpu_uuid:  # if the server reports which GPU served the request
                s.set_attribute("ws.gpu.uuid", out.gpu_uuid)

        with tracer.start_as_current_span("detokenize"):
            return tokenizer.decode(out.token_ids)

The GPU itself does not emit spans. We join the trace to the hardware through the resource attributes the agent adds (node, rack) and, where the server reports it, the GPU UUID, then use Grafana's trace-to-metrics links to jump from a slow span to that GPU's DCGM panels and that node's fabric ports for the same minute. Request-latency histograms carry exemplars, so a spike on a latency panel links straight to an example trace.

For training, a span per step is far too many. We trace the coarse phases (job start, data loading, each checkpoint save and restore, evaluation) and record step time as a histogram metric with the job label. A sudden rise in step time with flat GPU utilisation and rising link-error recovery on the job's ports is the classic signature of a fabric fault, and the job-to-fabric join in section 11 finds the port.

15. Cardinality and retention

Cardinality, the number of distinct label combinations, drives memory, storage and query cost. AI clusters are well behaved if you are deliberate about it. A worked example for a 32-node, 256-GPU cluster with a rail-optimised InfiniBand compute fabric (about 1,000 ports counting both host and switch ends):

Source	Series	Interval	Samples per second
UFM fabric counters, 8 kept counters × ~1,000 ports	~8,000	60 s	~135
Same, if every counter in a debug counter set is streamed (~100)	~100,000	60 s	~1,670
DCGM, ~27 default fields × 256 GPUs	~6,900	15 s	~460
node-exporter after the keep list, ~300 × 40 nodes	~12,000	15 s	~800
Collectors, exporters, UFM system, services	~5,000	15–60 s	~200
Total (kept counters)	~32,000		~1,600

At roughly 1.5 bytes per compressed sample, 1,600 samples a second is about 75 GB of object storage a year. That is small enough that we keep raw data for 13 months for clusters of this size; Mimir's retention is per tenant, so very large fleets put long-term recording rules in a separate tenant with longer retention than the raw data.

The things that break budgets are labels, not metrics:

Job IDs on fabric or node series. A busy Slurm cluster runs thousands of jobs a day; each new value creates new series. Keep hpc_job on GPU series only, and join through ws_job_node_info.
Pod UIDs, container IDs, request IDs and trace IDs as labels. Drop them in the gateway; trace IDs belong in exemplars.
Per-lane and per-link counters everywhere. Enable NVLink per-link error counters and per-lane fabric counters for diagnosis, scoped to the nodes under investigation, not by default.
Raw GUIDs as well as names. Use the labelled UFM endpoint and keep names; one identity label per object is enough.

Retention we start from, and adjust per customer:

Data	Store	Retention
Metrics (raw)	Mimir	13 months for clusters up to a few hundred GPUs
Metrics (local, if Prometheus is used)	Prometheus	15 days
Traces	Tempo	14 days, after tail sampling (all errors, all slow requests, 10% of the rest)
Fabric and host logs and events	Elasticsearch	hot 7 days, warm to 30, cold to 90, then delete
Audit and security logs	Elasticsearch	As the customer's policy requires, often a year or more
Topology snapshots and NDT reports	Elasticsearch or object storage	Life of the cluster

16. High availability of the observability stack

Monitoring that fails at the same time as the thing it monitors is decoration. The design places every stateful component across three failure zones (two management racks on separate power feeds and a small third zone for quorum):

UFM: active/standby across the two racks; everything talks to the virtual IP. Plugins are loaded on both nodes.
Collector gateways: at least two, both active. Agents use the pool name and a persistent queue, so a gateway loss is a retry, not a gap. Fabric scrapes run on both and are deduplicated in Mimir.
Mimir: zone-aware replication with a replication factor of 3, and the HA tracker for the fabric scrapes and any Prometheus pairs.
Alertmanager: three peers in gossip so each notification goes out once, plus an external dead man's switch fed by the Watchdog alert.
Elasticsearch: three master-eligible nodes (one voting-only in the quorum zone), one replica per shard, shard allocation awareness on the zone attribute, and snapshots to object storage.
Grafana and Kibana: two instances each, stateless behind a load balancer; Grafana uses a shared database, and both are provisioned from Git so a rebuild is a deploy.

The configuration that matters, in Mimir:

# mimir.yaml (excerpt)
distributor:
  ha_tracker:
    enable_ha_tracker: true
    kvstore:
      store: etcd
      etcd:
        endpoints: ["etcd-1.obs.internal:2379", "etcd-2.obs.internal:2379", "etcd-3.obs.internal:2379"]

limits:
  accept_ha_samples: true
  ha_cluster_label: cluster
  ha_replica_label: replica
  compactor_blocks_retention_period: 395d

ingester:
  ring:
    replication_factor: 3
    zone_awareness_enabled: true
    # set per instance: zone-a, zone-b or zone-c
    instance_availability_zone: zone-a

And in Elasticsearch, per node:

# elasticsearch.yml (zone A hot node; the zone C node uses node.roles: [ master, voting_only ])
cluster.name: obs-ai-prod
node.roles: [ master, data_hot, data_content, ingest ]
node.attr.zone: a
cluster.routing.allocation.awareness.attributes: zone
discovery.seed_hosts: [ "es-hot-1.obs.internal", "es-hot-2.obs.internal", "es-quorum-1.obs.internal" ]

The monitoring of the monitoring runs out of band: a small separate Prometheus (or a hosted check) watches the gateways, Mimir, Elasticsearch health and the dead man's switch, so a total loss of the main stack is itself detected.

17. Proving it works: fault drills at handover

We do not hand over a dashboard we have not seen go red. For every new cluster we run, and record:

Pull a cable between a leaf and a spine during a test job: the flap alert pages within the expected window, UFM raises an event, the event is in Kibana, and the job-to-fabric view shows which job crossed that port.
Fail the active UFM server: the standby takes the virtual IP, scrapes resume, and the gap in fabric data is shorter than the alert window.
Stop a gateway: no data loss, a GatewayRedundancyLost page, and recovery when it returns.
Stop an Elasticsearch node and a Mimir zone: queries still answer, writes continue, health alerts fire.
Run PDR in dry-run against a deliberately marginal cable on a lab port: the "would isolate" decision appears in the log store.
Compare as-built with design using the NDT plugin and file the report with the handover pack.

18. How Workstation delivers and runs this

Workstation is an NVIDIA partner, so alongside the hardware we provide engineering, software and staff to set up and run AI data centres end to end: design, installation, fabric bring-up, UFM and observability set-up, and the engineers who operate it with you afterwards. NVIDIA's own overview of UFM is on the official NVIDIA Unified Fabric Manager (UFM) page.

Supply: GPU systems, InfiniBand and Ethernet switches, adapters, optics and UFM licences, specified together.
Build: UFM HA, plugins, Slurm or Kubernetes integration, DCGM and dcgm-exporter, Collector agents and gateways, and either our hosted backends or yours, all from versioned configuration.
Run: managed observability around the clock: alert tuning, upgrades (including migrations like the dcgm-exporter 4.9.0 renames), capacity planning for the monitoring stack, and a monthly report on GPU efficiency, fabric faults and cost per useful GPU-hour.

For a sizing conversation, ask our AI Sales Engineer or browse the networking and DGX ranges in the shop.

19. Sources and verification notes

NVIDIA UFM overview and tiers: the official NVIDIA Unified Fabric Manager (UFM) page.
UFM SDK plugins, ports, paths, counter names and PDR defaults: the READMEs and dashboards in Mellanox/ufm_sdk_3.0, read in November 2026. Plugin behaviour changes between releases; check the README for the version you deploy.
dcgm-exporter metric names and the 4.9.0 renames: etc/default-counters.csv and the 4.9.0 release notes in NVIDIA/dcgm-exporter.
OpenTelemetry Collector components are from the contrib distribution; configuration keys move between releases, so validate with otelcol-contrib validate --config=... before deploying.
UFM tier packaging, NetQ and DOCA Telemetry Service details are described generally; consult NVIDIA's current documentation for exact features and licensing for your versions.
Counter names marked ws: or ws_ are Workstation conventions (recording rules and info metrics), not NVIDIA metrics.

Dashboard gallery · example data

Seventeen dashboards for an NVIDIA AI cluster

The views we build for a typical 32-node, 256-GPU cluster. Open a panel to see a mock of the dashboard, the queries behind it and the alert thresholds we start from. Metric names are the ones in the technote: UFM telemetry counters, dcgm-exporter 4.9.0 fields, and our own ws: recording rules where the source varies.

Ports reporting, aggregate bandwidth by topology level, error recoveries per switch port and active UFM alarms, scraped from the labelled UFM telemetry endpoint every 60 seconds.

Grafana · Fabric health · cluster=ai-prod-1 · fabric=ib-computeExample data

1,022 / 1,024ports reporting

99.8%links with zero errors (24 h)

3active UFM alarms

ufm-aactive UFM server (HA)

Aggregate bandwidth by level, Tb/s (24 h)host → leafleaf → spine

Error recoveries per port, last hour (8 leaves × 32 ports)

leaf-1 … leaf-8 (rows)red = recoveries in the last hour

Switch	Port	Peer	Recoveries 1 h	Link downs 24 h	State
leaf-r07-c	18	spine-2 p41	6	2	flapping
leaf-r09-f	5	gpu-r09-02 rail 4	3	0	degrading
leaf-r08-a	29	spine-1 p12	1	0	watch

Queries

# bandwidth by topology level (bits per second), as in NVIDIA's dashboard
sum by (level, peer_level) (
  rate(PortXmitDataExtended{level!="", peer_level!=""}[5m]) * 8
)

# ports reporting right now
count(PortXmitDataExtended{fabric="ib-compute"})

# worst ports by error recovery in the last hour
topk(10, increase(LinkErrorRecoveryCounterExtended[1h]))

Alert thresholds

pageUFM telemetry target down for 5 minup{job="ufm-telemetry"} == 0
pageNo fabric counters for 10 minabsent_over_time(PortXmitDataExtended[10m])
ticketPorts reporting below expected by more than 0.5%compared with the cabling plan

Link-downed, error-recovery, link-integrity and buffer-overrun counters as rates and as a timeline, joined to the node on the other end of each link through the ws_ib_link_info map built from UFM's links resource.

Grafana · Port errors · last 24 hExample data

4link-down events (24 h)

2ports with integrity errors

0buffer overruns

1port isolated by PDR

Link-down events per hour

Error recoveries, leaf-r07-c port 18 (cumulative)count

Time	Switch / port	Event	Far end	Jobs crossing
13:42	leaf-r07-c / 18	link down, up after 9 s	spine-2 / 41	3
13:17	leaf-r07-c / 18	link down, up after 7 s	spine-2 / 41	3
10:05	leaf-r09-f / 5	integrity errors	gpu-r09-02 mlx5_4	1

Queries

# flaps: two or more link-downs in 15 minutes
increase(LinkDownedCounterExtended[15m]) >= 2

# retrains and physical-integrity errors
increase(LinkErrorRecoveryCounterExtended[15m]) > 0
increase(LocalLinkIntegrityErrorsExtended[15m]) > 0
increase(ExcessiveBufferOverrunErrorsExtended[15m]) > 0

# which node is on the far end
increase(LinkDownedCounterExtended[1h])
  * on (switch, port) group_left (node) ws_ib_link_info

Alert thresholds

pageLink flapping2 or more link-downs in 15 min
ticketError recovery or integrity errorsany increase in 15 min: check cable and transceiver
ticketBuffer overrunsany increase in 15 min

Uplink utilisation per leaf over time, UFM's congestion bandwidth figure by topology level, and the busiest ports. Congestion is a capacity and placement signal, so it raises tickets, not pages.

Grafana · Congestion · last 24 hExample data

84%p95 uplink use, leaf-r08 (busiest)

58%p95 uplink use, fabric average

0.12max Normalized_CBW, leaf → spine

Uplink utilisation by leaf (rows) per hour (columns)

00:00darker = busier, red = over 80%23:00

Max Normalized_CBW by levelhost → leafleaf → spine

Queries

# UFM's congestion bandwidth figure, worst by level
max by (level, peer_level) (Normalized_CBW{level!="", peer_level!=""})

# uplink utilisation: our recording rule over port speed
# (ws_ib_port_speed_bps comes from the UFM links exporter)
  ws:ib_port_tx_bps:rate5m{port_role="uplink"}
/ on (switch, port) group_left ws_ib_port_speed_bps

# credit wait: the transmit-wait counter's name depends on the
# counter set you enable; list it first with
#   curl -s http://ufm:9001/csv/metrics | head -1

Alert thresholds

ticketUplink p95 above 80% for 30 mincapacity or job-placement review
ticketCongestion trend up week on weekreviewed in the monthly report
infoCheck units firstconfirm Normalized_CBW semantics for your UFM version before alerting on it

Device temperature against PDR's 70 °C default, error recoveries over seven days per link, and the replacement queue. A large fabric has thousands of optics; a handful degrading each quarter is normal, and catching them early is the point.

Grafana · Cables and optics · last 7 daysExample data

2,048link ends monitored

1device over 70 °C

3links with a rising error trend

1RMA open

dev_temperature, leaf-r09-f (°C, 7 days)temperature70 °C

Error recoveries per day, three degrading links

Link	Temp °C	Recoveries 7 d	PDR	Action
leaf-r09-f/5 ↔ gpu-r09-02	73	20	isolated	swap optic, Tue window
leaf-r07-c/18 ↔ spine-2/41	61	14	would isolate	reseat, then replace
leaf-r08-a/29 ↔ spine-1/12	59	9	healthy	watch

Queries

# hot devices (PDR's default TMAX is also 70 C)
dev_temperature > 70

# links that retrained in the last 7 days, worst first
sort_desc(increase(LinkErrorRecoveryCounterExtended[7d]) > 0)

# trend: this week against last week
  increase(LinkErrorRecoveryCounterExtended[7d])
- increase(LinkErrorRecoveryCounterExtended[7d] offset 7d)

Alert thresholds

ticketDevice above 70 °C for 10 minmatches PDR TMAX; PDR also watches a 10 °C rise
ticketError recoveries rising week on weekschedule a replacement
infoOptical power and BER detailcome from UFM cable data and PDR; exposure varies by version

PDR's decisions come from its log, shipped to Elasticsearch, alongside the counters that drove them. During the dry-run weeks the "would isolate" column is the one to review.

Kibana + Grafana · PDR · last 30 daysExample data

Livemode (after 3 weeks dry run)

1port isolated now

7would-isolate decisions in dry run

5returned to service after repair

Time	Port	Reason	Decision	Ticket
08 Nov 02:14	leaf-r09-f/5	temperature over TMAX	isolated	open
05 Nov 19:40	leaf-r03-b/22	BER over 125-minute threshold	de-isolated	closed: optic replaced
02 Nov 11:02	leaf-r07-c/18	link down on both ends	dry run	reviewed

Queries

# KQL: PDR decisions in the UFM event stream
data_stream.dataset : "ufm.events" and message : (*isolat* or *deisolat*)

# PromQL: the link-down signal PDR also uses
increase(LinkDownedCounterExtended[15m]) > 0

# exclusion list held by the plugin's REST API
GET  /excluded
PUT  /excluded     [["<port-guid>_<port>", <ttl-seconds>]]

Alert thresholds

pageAny isolation in live modebounded by MAX_NUM_ISOLATE (default 10, or 0.5% of the fabric)
ticketEvery dry-run "would isolate"reviewed against counters
infoPDR defaultsINTERVAL 300 s · TMAX 70 °C · D_TMAX 10 · MAX_PDR 1e-12 · DEISOLATE_CONSIDER_TIME 5 min

PFC pause frames, ECN marking and drops on the RoCE traffic class, plus routing-session health. Counter names differ between NetQ, gNMI and OTLP export, so the panels use our normalised ws:eth_* recording rules.

Grafana · Ethernet fabric · fabric=eth-storageExample data

0drops on lossless class (1 h)

18k/sPFC pause frames received, peak

2.1M/sECN-marked packets, peak

128 / 128BGP sessions established

PFC pause rx and ECN marked (normalised, 6 h)ECNPFC

Discards per switch (1 h)

Queries

# our normalised rules (source counters vary by export path)
sum by (switch) (ws:eth_pfc_pause_rx:rate5m{fabric="eth-storage"})
sum by (switch) (ws:eth_ecn_marked:rate5m{fabric="eth-storage"})
sum by (switch, traffic_class) (ws:eth_drops:rate5m)

# SNMP fallback (IF-MIB via snmp_exporter)
sum by (instance, ifName) (rate(ifHCInOctets[5m]) * 8)
sum by (instance, ifName) (rate(ifOutDiscards[5m]))

Alert thresholds

pageDrops on the lossless (RoCE) classany, sustained 5 min
pagePFC stormpause rate far above baseline on one port for 5 min
ticketECN marking climbing with flat throughputcongestion-control tuning review

DPUs scraped from the fabric collector through a file-based target list, labelled with the host each DPU serves. Field names depend on which DTS providers you enable, so we start from the target-level view and add counters per site.

Grafana · BlueField · 64 DPUsExample data

64 / 64DPUs reporting

0.002%rx drop ratio, worst DPU

71 °Chottest DPU

Scrape success per DPU (rows of 16)

green-ish = reportingred = missing

Queries

# every DPU target is up
count(up{job="doca-telemetry"} == 1)

# DPUs missing, with the host they serve
up{job="doca-telemetry"} == 0

# scrape cost per DPU (watch growth as providers are added)
scrape_samples_scraped{job="doca-telemetry"}

Alert thresholds

ticketDPU target down for 10 minpage if it serves storage or the control plane
ticketSeries per DPU up 50% after a changecardinality guard

Every GPU in one heatmap, using the graphics-engine and tensor-pipe profiling fields rather than the basic utilisation gauge, which reads close to full even while a GPU waits on the network.

Grafana · GPU fleet · 32 nodes × 8 GPUsExample data

256GPUs reporting

92%allocated to jobs

71%graphics engine busy

48%tensor pipe active

63%GPU memory used

Graphics engine busy per GPU (nodes are columns)

gpu-r01-01 … gpu-r08-04pale = idle

Fleet average, 24 hgraphics enginetensor

Queries

# dcgm-exporter 4.9.0 names (ratios, 0 to 1)
avg(DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO)
avg(DCGM_FI_PROF_TENSOR_UTIL_RATIO)

# memory used across the fleet
sum(DCGM_FI_DEV_FB_USED_BYTES) / sum(DCGM_FI_DEV_FB_CAPACITY_BYTES)

# share of GPUs held by a job
count(DCGM_FI_DEV_GPU_UTIL_RATIO{hpc_job!=""}) / count(DCGM_FI_DEV_GPU_UTIL_RATIO)

Alert thresholds

ticketJob holds GPUs at under 5% for 30 minGPUAllocatedButIdle
pagedcgm-exporter down for 5 minup{job="dcgm"} == 0
infoMonthly reviewgraphics engine busy under 60% of allocated time

Temperatures, board power and energy, XID events and memory row remapping. XIDs are the GPU driver's error codes; the table links each one to its runbook.

Grafana · GPU health · last 24 hExample data

78 °Chottest GPU

251 kWGPU board power, fleet

2XID events (24 h)

0row-remap failures

GPU board power, kW (24 h)

Max GPU temperature per rack, °C

Time	Node / GPU	XID	Meaning (summary)	Action
06:51	gpu-r05-02 / 6	94	contained ECC error	job restarted; watch
03:12	gpu-r02-04 / 1	79	GPU no longer on the bus	drained; RMA case

Queries

max by (rack) (DCGM_FI_DEV_GPU_TEMP_CELSIUS)
sum(DCGM_FI_DEV_BOARD_POWER_WATTS) / 1000

# energy over a day, kWh (joules counter in 4.9.0)
sum(increase(DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL[1d])) / 3.6e6

# new XID values
changes(DCGM_FI_DEV_XID_ERROR[10m]) > 0 and DCGM_FI_DEV_XID_ERROR != 0

DCGM_FI_DEV_ROW_REMAP_FAILED > 0

Alert thresholds

pageNew XID on a GPUwith a runbook per XID family
pageRow remapping faileddrain the node and raise an RMA
ticketGPU over 85 °C for 10 minset per platform: air and liquid cooling differ
ticketMore than 100 PCIe replays in 15 minDCGM_FI_DEV_PCIE_REPLAY_TOTAL

NVLink throughput per GPU from the default field set, and the optional CRC, replay and recovery error counters we enable in the counters CSV. On rack-scale NVL72 systems the NVLink switch fabric is labelled as its own fabric="nvlink".

Grafana · NVLink · 32 nodesExample data

1.1 TB/speak NVLink per GPU (1 h)

0CRC flit errors (24 h)

12replay errors, gpu-r06-01

NVLink throughput per GPU (nodes are columns)

GPU 0 … 7 (rows)darker = more traffic

Queries

sum by (node) (DCGM_FI_DEV_NVLINK_THROUGHPUT_BYTES_PER_SECOND)

# optional fields (enable in the counters CSV); Blackwell and newer
increase(DCGM_FI_DEV_NVLINK_RECOVERY_EVENT_TOTAL[1h]) > 0
increase(DCGM_FI_DEV_NVLINK_RECOVERY_FAILED_TOTAL[1h]) > 0
# Hopper: DCGM_FI_DEV_NVLINK_REPLAY_TOTAL, DCGM_FI_DEV_NVLINK_RECOVERY_TOTAL

Alert thresholds

ticketAny NVLink recovery events in 1 h; page on a failed recoverycheck against the job's step time
ticketNVLink silent on a busy GPUpossible topology or driver issue

Storage-fabric throughput from the hosts' InfiniBand counters, file-system latency from the storage vendor's exporter, and GPUDirect Storage activity from the nvidia-fs driver statistics through a textfile collector. Vendor metric names vary, so they are mapped to ws:fs_* rules.

Grafana · Storage · fabric=ib-storageExample data

182 GB/sread, peak (1 h)

41 GB/swrite, peak (checkpoints)

9.4 msp99 read latency

38%reads via GPUDirect Storage

Read and write, GB/s (6 h)readwrite

Queries

# storage-rail HCAs on each host (node-exporter infiniband collector)
sum(rate(node_infiniband_port_data_received_bytes_total{fabric="ib-storage"}[5m]))
sum(rate(node_infiniband_port_data_transmitted_bytes_total{fabric="ib-storage"}[5m]))

# vendor exporter, normalised
histogram_quantile(0.99, sum by (le) (rate(ws:fs_read_latency_seconds_bucket[5m])))

# nvidia-fs stats via textfile collector (our convention)
sum(rate(ws_nvfs_read_bytes_total[5m])) / sum(rate(ws:fs_read_bytes:total[5m]))

Alert thresholds

ticketp99 read latency above 20 ms for 10 mintune per file system
pageStorage fabric link flappingsame rules as the compute fabric
ticketCheckpoint writes taking over twice baselinefrom the job's checkpoint spans

Pick a job ID; the dashboard finds its GPUs through dcgm-exporter's hpc_job label, its nodes through ws_job_node_info, and its switch ports through ws_ib_link_info. This is the view that ends the "is it the network?" argument.

Grafana · Job 4471823 · account=genomics · 8 nodesExample data

64GPUs on 8 nodes

82%graphics engine busy

+18%step time against first hour

1job port with error recoveries

Node	GPU busy	Leaf / port (rail 3)	Recoveries 1 h	Verdict
gpu-r07-01	84%	leaf-r07-c / 17	0	ok
gpu-r07-02	83%	leaf-r07-c / 18	6	fabric
gpu-r07-03	81%	leaf-r07-c / 19	0	ok
gpu-r08-01	80%	leaf-r08-c / 17	0	ok

Queries

avg(DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO{hpc_job="$job"})

# error recoveries on every port the job's nodes use
sum by (switch, port) (
  increase(LinkErrorRecoveryCounterExtended[1h])
    * on (switch, port) group_left (node)
      (ws_ib_link_info * on (node) group_left () ws_job_node_info{hpc_job="$job"})
)

# UFM-Slurm integration log, per job (KQL)
log.file.path : "/var/log/slurm/ufm_slurm.log" and message : "4471823"

Alert thresholds

ticketError recoveries on a running job's portsannotate the job and tell its owner
ticketStep time up 15% with flat GPU busyclassic fabric signature

dcgm-exporter adds namespace, pod and container labels; kube-state-metrics gives the GPU requests. Together they show who asked for GPUs and who is using them.

Grafana · Kubernetes GPUs · cluster=ai-k8s-1Example data

184GPUs requested

176GPUs with a running pod

3GPU pods pending over 15 min

Namespace	GPUs requested	Avg busy	Memory used	Note
inference-prod	64	68%	81%	ok
research-llm	96	79%	92%	ok
notebooks	16	3%	22%	idle allocations
batch-eval	8	54%	40%	ok

Queries

sum by (namespace) (kube_pod_container_resource_requests{resource="nvidia_com_gpu"})

avg by (namespace) (DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO{namespace!=""})

sum by (namespace) (DCGM_FI_DEV_FB_USED_BYTES{namespace!=""})
  / sum by (namespace) (DCGM_FI_DEV_FB_CAPACITY_BYTES{namespace!=""})

Alert thresholds

ticketNamespace busy under 5% for 2 h with GPUs heldnotebook and dev namespaces
ticketGPU pods pending over 15 mincapacity or quota

Service-level latency from OpenTelemetry GenAI metrics with exemplars, and a sampled slow trace broken into its spans. From the slow span, Grafana links to that node's GPU and fabric panels for the same minute.

Grafana · chat-gateway · model=llm-70bExample data

420 mstime to first token, p95

21 mstime per output token, p50

340/srequests

0.2%errors

Time to first token p95, ms (6 h)p95SLO 1,000 ms

Sampled slow trace, 1.84 s

chat.completions

tokenize

model_server.generate

· queue

· prefill

· decode

detokenize

Queries

# OpenTelemetry GenAI server metrics, as Prometheus names
histogram_quantile(0.95, sum by (le, gen_ai_request_model) (
  rate(gen_ai_server_time_to_first_token_seconds_bucket[5m])))

histogram_quantile(0.5, sum by (le) (
  rate(gen_ai_server_time_per_output_token_seconds_bucket[5m])))

# TraceQL: slow requests for one service
{ resource.service.name = "chat-gateway" && duration > 2s }

Alert thresholds

pageTTFT p95 over 1 s for 10 minthe service-level objective
pageError rate over 1% for 5 minfrom http.server.request.duration
ticketQueue span over 30% of request timescale replicas or batch size

UFM events arrive through the syslog streaming plugin, switch syslog through the Collector or Logstash, and host and Slurm logs through the agents, all with the same site, cluster and fabric fields as the metrics.

Kibana · Discover · logs-* · last 24 hExample data

Fabric link events per hour

@timestamp	data_stream.dataset	host.name	message
13:42:07	network.switch	leaf-r07-c	port 18 link down
13:42:16	network.switch	leaf-r07-c	port 18 link up
13:42:18	ufm.events	ufm-a	link went down and came back (leaf-r07-c/18)
13:43:02	slurm	gpu-r07-02	NCCL timeout warning, job 4471823

Queries (KQL)

data_stream.dataset : "network.switch" and tags : "fabric_link_event"

data_stream.dataset : "ufm.events" and log.level : ("critical" or "error")

cluster : "ai-prod-1" and host.name : gpu-r07-* and message : *NCCL*

Alert thresholds

ticket5 or more link events on one switch in 10 minKibana rule, grouped by host.name
pageUFM critical eventUFM alarm severity critical
infoRetentionhot 7 days, warm to 30, cold to 90

Installed GPU-hours, allocated GPU-hours and busy GPU-hours, priced with a fully loaded cost per installed GPU-hour that finance supplies (here an illustrative £2.50). The gap between the first and last is the money fabric and scheduling problems cost.

Grafana · Capacity and cost · OctoberExample data

190,464installed GPU-hours (256 × 744)

175,227allocated (92%)

135,229busy (71%)

£3.52cost per busy GPU-hour

186 MWhGPU energy

Busy GPU-hours by team, thousands

research-llm · genomics · inference · vision · eval · notebooks

Queries

# recording rules, evaluated every 5 minutes
- record: ws:gpus_installed:count
  expr: count(DCGM_FI_DEV_GPU_UTIL_RATIO)
- record: ws:gpus_busy:sum
  expr: sum(DCGM_FI_PROF_GR_ENGINE_UTIL_RATIO)

# busy GPU-hours over the month (in Grafana, $__range)
sum_over_time(ws:gpus_busy:sum[30d:5m]) / 12

# GPU energy, MWh
sum(increase(DCGM_FI_DEV_GPU_ENERGY_JOULES_TOTAL[30d])) / 3.6e9

# cost per busy GPU-hour = installed hours x cost / busy hours
(744 * 256 * 2.50) / 135229   # = 3.52

Alert thresholds

reviewBusy under 60% of installedmonthly capacity review
reviewTeam with allocation far above usequota conversation
infoEvery 1% of utilisationabout £56,000 a year at 256 GPUs and £2.50

The monitor's own health, scraped by a small out-of-band Prometheus so that a total loss of the main stack is still noticed. Every row shows redundancy left, not just up or down.

Grafana (out of band) · Observability platformExample data

ufm-aUFM active; ufm-b standby, in sync

2 / 2collector gateways

9 / 9Mimir ingesters, 3 zones

greenElasticsearch, 3 masters

3 / 3Alertmanager peers

Component	Zone A	Zone B	Zone C	Can lose
UFM Enterprise	active	standby	—	1 server
OTel gateway	up	up	—	1 gateway
Mimir	3 ingesters	3 ingesters	3 ingesters	1 zone
Elasticsearch	master + hot	master + hot	voting only	1 zone
Alertmanager	peer	peer	peer	1 peer

Queries

count(up{job="otel-gateway"} == 1)

# Mimir ingesters active in the ring
sum(cortex_ring_members{name="ingester", state="ACTIVE"})

max(elasticsearch_cluster_health_status{color="green"})
max(alertmanager_cluster_members)

# collector pipeline health
sum by (instance) (rate(otelcol_exporter_send_failed_metric_points_total[5m]))
max by (instance) (otelcol_exporter_queue_size / otelcol_exporter_queue_capacity)

# UFM API reachable through the virtual IP
up{job="ufm-system"}

Alert thresholds

pageGateway redundancy lost for 5 minfewer than 2 gateways up
pageElasticsearch red for 2 minyellow raises a ticket
pageExporter failing for 10 mindata is queuing, not arriving
pageWatchdog missingexternal dead man's switch
ticketUFM failover happenedcheck why the active server failed

All values are illustrative example data drawn for this page. Queries use the metric names described in the technote; names marked ws: or ws_ are Workstation recording rules and info metrics. Thresholds are starting points we tune per site.

UFM tiers at a glance

Four things to remember

NVIDIA UFM for AI Clusters: Why InfiniBand Fabric Visibility Decides What Your GPUs Are Worth

Why the fabric decides what your GPUs are worth

What NVIDIA UFM is, in plain terms

The three UFM tiers and when each fits

Three clusters, three answers

1. A four-node DGX B300 or HGX B300 cluster

2. A 32-node SuperPOD-style cluster

3. A university HPC service with Slurm

What good fabric visibility gives you

Monitor the whole stack, not just the fabric

What Workstation does

Questions to ask before you buy

NVIDIA partner: engineering, software and people

Next steps

NVIDIA Observability with OpenTelemetry: UFM SDK Plugins, DCGM, DOCA and NetQ into Mimir, Grafana and the Elastic Stack

1. The signals that matter in an AI cluster

2. NVIDIA UFM: tiers, telemetry endpoints and REST API

Telemetry endpoints

REST API

High availability

3. The UFM SDK and its plugins

4. Problem Detection and Resolution in practice

5. GPU telemetry: DCGM and dcgm-exporter

dcgm-exporter 4.9.0 renamed the default metrics

Which GPU metric means "busy"?

Job and pod labels

NVLink and NVSwitch

6. BlueField DPUs and Spectrum-X Ethernet

7. Workstation's OpenTelemetry reference design

8. Collector agent configuration

9. Gateway configuration and the two backends

10. Prometheus scrape configuration (if you run Prometheus)

11. Fleet-wide labelling scheme

12. Alert rules

13. Logs and events into the Elastic stack

Fluentd: UFM event and telemetry streams to Elasticsearch

Logstash: switch syslog

Retention with index lifecycle management

14. Tracing AI inference and training services

15. Cardinality and retention

16. High availability of the observability stack

17. Proving it works: fault drills at handover

18. How Workstation delivers and runs this

19. Sources and verification notes

01InfiniBand fabric healthThe first screen on-call opens: is the fabric whole?UFM Enterprise

Queries

Alert thresholds

02Port errors and link flapsEvery link that went down, retrained or overranUFM Telemetry

Queries

Alert thresholds

03Congestion and credit waitIs a hot spot holding the collective back?UFM Telemetry

Queries

Alert thresholds

04Cables and transceiversFind the optic that is getting worse, before it failsUFM · PDR

Queries

Alert thresholds

05PDR predictive maintenanceWhat Problem Detection and Resolution isolated, and whypdr_deterministic_plugin

Queries

Alert thresholds

06Spectrum-X Ethernet and RoCELossless behaviour on the Ethernet fabricsNetQ · switch telemetry

Queries

Alert thresholds

07BlueField DPUsPort and health counters from DOCA Telemetry ServiceDOCA DTS

Queries

Alert thresholds

08GPU fleet utilisationAllocated is not the same as busyDCGM

Queries

Alert thresholds

09GPU thermals, power and XID errorsHardware health, per GPU and per rackDCGM

Queries

Alert thresholds

10NVLink and NVSwitchThe fabric inside each serverDCGM

Queries

Alert thresholds

11Storage: GPUDirect Storage and parallel file systemAre the GPUs waiting on data?node-exporter · FS exporter

Queries

Alert thresholds

12Slurm job view: job → GPU → fabricOne job, its GPUs and every switch port it crossesDCGM · UFM · Slurm