Inside a 32-GPU HGX B300 AI Factory: Compal SGX30-2 Nodes, a Rail-Optimised 800G Fabric and the Workflows It Runs

This tech note describes a 32-GPU NVIDIA HGX B300 AI factory built from four Compal SGX30-2 servers: the bill of materials, the architecture of one node, the four networks around it, the software stack, the workflows it runs from data to production, model-sizing maths, and the bring-up and acceptance tests we use before handover. For the business view without the commands, read the companion blog post.

1. Bill of materials

Per node, as configured:

Rack totals for four nodes:

One point to plan around: Blackwell Ultra trades double precision for AI throughput. NVIDIA rates HGX B300 at 10 TFLOPS FP64 per board, against 296 TFLOPS for HGX B200. This rack is for AI work. If you also run FP64 simulation codes, put them on B200 or GB200 nodes.

2. Node architecture

2.1 GPUs and NVLink

The eight GPUs share fifth-generation NVLink through NVLink switches on the baseboard: 1.8 TB/s per GPU and 14.4 TB/s across the board. Inside a node, every GPU reaches every other at full NVLink speed, so tensor parallelism of up to 8 is cheap. Beyond 8 GPUs, traffic crosses the 800G fabric, which is roughly an order of magnitude slower per GPU than NVLink. That boundary is the main design constraint for how you split models (§5).

The NVLink fabric must be initialised before CUDA will use the GPUs. Install the NVIDIA Fabric Manager package matching your driver version and enable its service. If nvidia-smi -q does not show the fabric as completed, jobs will fail or fall back to slow paths.

2.2 One NIC per GPU

On HGX B300 the eight ConnectX-8 SuperNICs sit on the GPU baseboard, each paired with one GPU, and each exposes one 800G OSFP cage at the front of the SGX30-2. ConnectX-8 includes its own PCIe switch, so the GPU-to-NIC path for GPUDirect RDMA is short and does not cross the CPU. Each port runs either InfiniBand XDR at 800 Gb/s or Ethernet as 2 × 400GbE (for Spectrum-X). Confirm NIC-to-GPU affinity with nvidia-smi topo -m: each GPU should show its NIC on the closest path.

2.3 CPUs and memory

The Xeon 6 6700P parts listed (6767P, 6776P, 6768P) are 64-core processors with eight DDR5 channels each. Thirty-two DIMMs therefore means two DIMMs per channel. That configuration prioritises capacity (3 TB, more than the 2.1 TB of HBM) over peak memory speed: two DIMMs per channel usually runs below the DIMMs' 6400 MT/s rating, so check the speed the BIOS reports and do not assume 6400. The capacity earns its keep in data loading, CPU offload of optimiser state during fine-tuning, and KV-cache offload for long-context serving.

Set the BIOS for performance (power profile, no deep C-states on serving nodes) and keep NUMA in mind. Pin data-loader processes to the socket closest to the GPUs they feed.

2.4 Storage

Eight Gen5 E1.S drives give 30.72 TB raw per node. We use them as a local tier, not as the system of record:

• Dataset staging: shards copied from shared storage before a run, so training reads at local NVMe speed.
• Checkpoint burst buffer: checkpoints land locally first, then drain asynchronously to shared storage, which keeps the GPUs from waiting on the network.
• Model cache: serving containers (NIM, vLLM, TensorRT-LLM engines) load weights from local disk, which cuts cold-start time.

A simple layout is one RAID 0 or striped LVM volume across the eight drives (XFS or ext4), treated as scratch, with anything precious replicated to shared storage. The two M.2 drives are mirrored for the OS.

2.5 BlueField-3 and management

The BlueField-3 B3240 carries storage and front-end traffic so it never shares a port with GPU collectives. It can also enforce network isolation for multi-tenant clusters and offload encryption. The BMC sits on a separate out-of-band network for power control, firmware and sensor telemetry.

2.6 Power and cooling

Twelve 3200 W Titanium supplies in 6+6 means each node survives the loss of a whole A or B feed. NVIDIA rates its own 8-GPU Blackwell Ultra system, DGX B300, at about 14 kW. Use that as the planning figure per node, and around 60 kW for four nodes, then confirm with power measurements during burn-in. The node is air cooled, so the hall needs high-density airflow with hot-aisle containment rather than a liquid loop.

3. Four networks

3.1 GPU fabric (east-west)

The fabric is rail-optimised: NIC n on every node connects to rail n. In an all-reduce, GPU n on each node mostly talks to GPU n on the other nodes, so traffic stays within one rail and one switch hop. Cross-rail traffic goes through NVLink inside the node instead of the network. NCCL is aware of this topology and uses it automatically when it is cabled consistently.

• One rack (32 ports): a single 64-port 800G switch, such as a Spectrum-X SN5600, or a Quantum-X800 InfiniBand switch hosts all eight rails with room for a second rack. With one switch, a "rail" is a logical grouping of ports, and the cabling discipline still matters for growth.
• A scalable unit (up to 32 nodes, 256 GPUs): dedicated leaf switches per rail group plus a spine layer, non-blocking. Reserve spine ports and cable pathways from day one.

InfiniBand versus Spectrum-X Ethernet is a choice of operations model more than raw speed: both run at 800 Gb/s per GPU here. InfiniBand has the longest track record for large training. Spectrum-X suits multi-tenant factories and teams that run Ethernet everywhere else. With Ethernet, enable RoCE with the NVIDIA-recommended congestion control settings. Default Ethernet settings are not good enough for AI collectives.

3.2 Storage network

Shared storage (a parallel file system or NVMe-over-Fabrics appliance) connects through the BlueField-3 ports on a separate network or VLAN. Size it for checkpoint bursts: four nodes writing a large model's full checkpoint at once is the peak to design for, not the average read rate.

3.3 Front-end network (north-south)

Users, applications and agents reach model endpoints through this network. We put WSL Proxy in front of every endpoint for TLS, authentication and web-application firewalling, and keep API keys, model-registry credentials and database passwords in WSL Vault, injected at runtime rather than baked into images.

3.4 Management network (out of band)

BMCs, switch management ports and PDUs go on an isolated management VLAN reachable only from admin jump hosts. Firmware updates, power cycling and sensor monitoring all happen here, so a misbehaving workload cannot lock you out of the hardware.

4. Software stack

5. Workflows: what the factory delivers

5.1 Ingest and prepare

Source data is pulled from document stores, ticketing systems, databases and code repositories into shared storage. Pipelines then de-duplicate, filter, redact personal data where policy requires, tokenise and shard it. For retrieval, documents are chunked and embedded on the GPUs and written to the vector database with their access-control metadata, so search results respect the same permissions as the source systems.

5.2 Train and fine-tune

Memory needed for full fine-tuning with an Adam-style optimiser in mixed precision is roughly 16 bytes per parameter before activations (weights, gradients and two optimiser moments plus a master copy), sharded across GPUs with FSDP or ZeRO-3:

Parameter-efficient methods such as LoRA need a small fraction of this, which is why most enterprise fine-tuning fits on one node. For parallelism, keep tensor parallelism inside a node (TP ≤ 8, on NVLink) and use data or pipeline parallelism across nodes (on the fabric). Checkpoint to the local E1.S tier, then drain to shared storage in the background.

5.3 Evaluate and promote

Every candidate model runs an evaluation suite (task accuracy on your own test sets, safety and refusal tests, latency and cost per request) before promotion. Ring Promoter moves the model and its serving configuration through int, test, acceptance and production rings, each with its own gates and approvals. A bad release is rolled back by promoting the previous version, not by hand-editing servers.

5.4 Serve

Weights-only memory is parameters × bytes per parameter (FP8 = 1 byte, NVFP4 ≈ 0.5 byte plus small scaling factors). On 288 GB GPUs:

The KV cache, not the weights, usually sets concurrency. Its size per token is 2 × layers × KV heads × head dimension × bytes. For Llama 3.1 405B (126 layers, 8 KV heads, head dimension 128) with an FP8 cache, that is 2 × 126 × 8 × 128 × 1 ≈ 258 KB per token, or about 33 GB for a single 128k-token conversation. That is why long-context serving benefits from big HBM, and from offloading cold KV blocks to the 3 TB of host DDR5 or to local NVMe, which Dynamo and recent vLLM versions support.

For high-traffic services, consider disaggregated serving: prefill (reading the prompt, compute-bound) and decode (writing the answer, memory-bandwidth-bound) on separate GPU pools, which Dynamo orchestrates. On a four-node rack a common starting split is one node for embeddings, retrieval and smaller models, two for the main assistant and agents, and one flexible node that serves by day and fine-tunes by night.

5.5 Retrieval and agents

Staff questions go through the front-end network to a retrieval service, which searches the vector database (filtered by the user's permissions), reranks the results and passes the best passages to the model with citations. Agents add tool calls: each tool is an API behind WSL Proxy, with credentials fetched from WSL Vault at call time and every call logged. Keep agents' permissions as narrow as a junior employee's, and log every action they take.

5.6 Operate and improve

DCGM health checks run continuously, with XID errors, ECC errors, NVLink errors, thermals and power alerting into SysOps247. Fabric telemetry from UFM or NetQ catches link flaps and congestion. Usage and quality signals (thumbs up and down, corrections, escalations) are captured and fed back as new training and evaluation data, which closes the loop in §5.1.

6. Bring-up and acceptance tests

We run these on every node and across the rack before handover, and record the results for your acceptance report.

6.1 Inventory and topology

nvidia-smi -L                          # 8 GPUs per node
nvidia-smi -q | grep -iA3 fabric       # NVLink fabric state: completed
nvidia-smi topo -m                     # GPU-NIC affinity and NVLink paths
nvidia-smi nvlink -s                   # all links up
ibstat | grep -E "State|Rate"          # 8 ports Active at 800 Gb/s (InfiniBand)
dmidecode -t memory | grep -c "Size: 96 GB"   # 32 DIMMs
nvme list                              # 8 x E1.S + 2 x M.2

6.2 GPU diagnostics and burn-in

dcgmi diag -r 3                        # extended diagnostics, every node
# then a 24-48 h soak with sustained GPU load while logging temperatures,
# clocks, power draw and XID/ECC events with dcgmi dmon

6.3 NCCL inside and across nodes

# single node, 8 GPUs over NVLink
./build/all_reduce_perf -b 8 -e 16G -f 2 -g 8

# four nodes, 32 GPUs over the fabric (one rank per GPU)
mpirun -np 32 -N 8 --hostfile hosts \
  -x NCCL_DEBUG=INFO -x NCCL_IB_HCA=mlx5 \
  ./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1

Compare the bus bandwidth at large message sizes against the reference figures for the platform and the fabric. A single slow rail shows up as a ceiling on the multi-node result. Check NCCL_DEBUG output to confirm NCCL found all eight NICs and is using GPUDirect RDMA.

6.4 Per-rail RDMA

# server on node-1, client on node-2, repeat for each NIC (rail 0..7)
ib_write_bw -d mlx5_0 --report_gbits -a
ib_write_bw -d mlx5_0 --report_gbits -a node-1

6.5 Storage

fio --name=seqread --filename=/scratch/fio --rw=read --bs=1M \
    --size=64G --numjobs=8 --iodepth=32 --direct=1 --group_reporting
fio --name=seqwrite --filename=/scratch/fio --rw=write --bs=1M \
    --size=64G --numjobs=8 --iodepth=32 --direct=1 --group_reporting

Then time a real checkpoint write and drain to shared storage. That number, not the synthetic one, tells you whether checkpoints will stall training.

6.6 End-to-end

Finally, serve a reference model through the full path (WSL Proxy, the serving stack, the GPUs) and run a load test at the expected number of concurrent users, recording latency to first token, tokens per second per user and total throughput. This becomes the baseline that SysOps247 alerts against.

7. Growing beyond one rack

Because the rack is built to a reference design, growth is repetitive rather than creative: add identical racks of four SGX30-2 nodes, extend the rails through leaf switches and a spine, and grow shared storage for checkpoint bandwidth. A scalable unit of 32 nodes (256 GPUs) is the natural next boundary. Beyond that, or if one model needs an NVLink domain larger than 8 GPUs, GB300 NVL72 racks can join the same storage, scheduler and front-end, as their own 72-GPU NVLink domains.

8. Supply, warranty and compliance

• Availability: the SGX30-2 is available to order only. HGX B300 is allocated by NVIDIA and the server makers, and lead times are measured in months. We confirm the date on a written quote.
• Warranty: 3-year return-to-base on the server, handled through us. On-site cover and local spares can be quoted.
• Export controls: advanced NVIDIA GPUs are controlled under UK and US rules. We need the end user, the intended use and the installation site before confirming supply.

Where to go next

• The business view: What an AI factory actually delivers.
• Platform pages: Compal SGX30-2, HGX B300 32-GPU rack, AI factory networking and all AI factory solutions.
• Order: Compal SGX30-2 in the shop.
• Questions about your own design? Talk to an engineer.