TuneBench: How Well Can AI Agents Calibrate Power System Models?

A new benchmark, TuneBench, asks whether LLM agents can calibrate power system dynamic models: adjusting controller parameters until a simulation reproduces measured step responses. Two frontier models did it very well. The headline figure, though, is a curve-fit statistic rather than an "accuracy" score, and the test design is the most reusable part of the paper. This post explains what was measured, what it does and does not show, and how to build the same kind of benchmark for your own agent work.

What TuneBench tests

Grid operators keep simplified "equivalent" dynamic models of generators and batteries for stability studies. Those models are only useful if their controller parameters make them respond like the real plant. Calibrating them by hand is slow, specialist work. TuneBench, published on arXiv on 3 November 2026 by Pomarico, Tessitore, Giannuzzi and Berizzi, frames that job as an agent task modelled on a transmission system operator's workflow of calibrating equivalent models against grid-code compliance tests [1].

The set-up, according to the paper [1]:

• Two cases. A synchronous generator with an IEEEG1 governor and an ESST1A automatic voltage regulator (AVR), and a battery energy storage system (BESS) using the WECC generic REGCA1, REECA1 and REPCA1 models.
• Two tests per case. Test P steps the active-power reference (0.5 to 0.7 pu at 5 s, then back down and up again); Test V steps the voltage reference to 0.97 pu. "pu" means per unit: values as a fraction of a rated base.
• Perturbed targets. Target parameters were deliberately moved away from standard IEEE and WECC defaults, so an agent cannot just recall textbook values.
• A hard budget. 15 simulations in total across both tests, split however the agent chooses, plus two verification runs that do not count. The random-search and genetic-algorithm (GA) baselines get 150.
• Repeats. Ten independent runs per method, with medians reported.
• Tools. The open-source ANDES simulator [2] is exposed through tool functions that run simulations, return waveforms and diagnostics, compute RMSE and R², and change parameters.

Five agents were tested: Opus 5.5, Sonnet 5, Haiku 4.5, GPT-5.6 and GPT-6 [1].

What "99.5%" actually means

Earlier coverage, including the first version of this post, described the result as "99.5% accuracy". That is not what the paper reports. The abstract says the two strongest agents reached a median R² above 0.995 in every test, with tracking errors up to two orders of magnitude lower than the baselines [1]. Four points matter.

• R² is a curve-fit statistic, not accuracy. R² (the coefficient of determination) compares the squared error between simulated and target trajectories with the variance of the target. It says how much of the target's movement the simulation explains, not what fraction of answers were right.
• R² saturates on easy tests. On the generator's active-power test, Sonnet 5, GPT-5.6 and the GA baseline all scored above 0.998 as well [1]. A high R² alone did not separate the methods there; RMSE (root-mean-square error, in pu) did.
• It applies to two models only. Haiku 4.5 failed most tests (R² of 0.03 on the generator voltage test), and Sonnet 5 and GPT-5.6 only marginally beat the baselines on the BESS power test (R² about 0.97) [1].
• The scope is narrow. Two cases, two tests each, a single device connected to an infinite bus, and parameter calibration of a known model structure. The authors list model identification, where the agent must also choose the control structure, as future work [1].

Values as stated in the paper's text [1]. "Group" means the paper gives one range for GPT-5.6, GA and random search together; "< 0.8" means the paper says only that all remaining methods were below 0.8. Not every figure is labelled as a median, so check the paper's tables before quoting any single cell.

Why the strongest agents won

The interesting finding is in the agent logs. The best agents did not search the parameter space blindly. They extracted conventional performance indices from the trajectories (settling time, ramp rate, oscillation frequency and damping), linked each to the parameters that govern it, and broke the problem into physically meaningful sub-problems [1]. That is how a control engineer works, and it is why 15 simulations were enough.

The weaker failures are instructive too. On the BESS voltage test the target was a lightly damped oscillation; most methods converged on overdamped responses, and the GA settled at the wrong steady-state voltage [1]. Haiku 4.5's best run on the generator power test settled near 0.55 pu instead of 0.7 pu, worse than the starting model [1]. A model that cannot reason about the physics falls back to guessing, and guessing with 15 tries loses to a GA with 150.

The cost was time. Median wall-clock time for Opus 5.5 was 22.1 minutes on the generator case and 67.5 minutes on the BESS case; GPT-6 took 17.6 and 21.7 minutes, and the authors judge it the best balance of accuracy and time [1]. The paper does not report token usage, monetary cost or baseline runtimes, so a cost comparison with the GA is not possible from the published data.

How the benchmark keeps agents honest

The companion repository shows a pattern worth copying [3]. ANDES is exposed to the agent as an MCP (Model Context Protocol) server. The simulation budget is not a request in the prompt: the server caps time-domain simulation calls through an environment variable (ANDES_MCP_MAX_TDS, set to 17 for 15 tuning runs plus two final checks), and writes a JSON Lines audit log that the README recommends keeping outside the agent's working folder [3]. Operating rules and permissions sit in the agent's configuration files rather than in conversation, and a machine-readable file defines permitted changes, protected files and reporting requirements [1][3].

This matters beyond power systems. An agent that is told "you have 15 runs" can miscount or ignore it. An agent whose tool refuses the sixteenth call cannot.

import json
import time


class SimulationBudget:
    """Hard cap enforced inside the tool, with an append-only audit log."""

    def __init__(self, limit, audit_path):
        self.limit = limit
        self.used = 0
        self.audit_path = audit_path  # keep outside the agent's writable folder

    def run(self, simulate, params):
        if self.used >= self.limit:
            raise RuntimeError(f"simulation budget of {self.limit} exhausted")
        self.used += 1
        result = simulate(params)
        with open(self.audit_path, "a", encoding="utf-8") as log:
            log.write(json.dumps({"ts": time.time(), "run": self.used, "params": params}) + "\n")
        return result

A playbook for benchmarking your own calibration agent

The same design works for any task where an agent tunes parameters against a simulator or test harness: controller tuning, digital-twin calibration, load-model fitting, even autoscaler or rate-limit settings against a replayed workload.

1. Write targets you can score. Record or generate target trajectories for a small set of standard tests, as TuneBench does with power and voltage steps.
2. Move the answer away from the defaults. Perturb target parameters from published values so recall cannot pass the test [1].
3. Fix the budget in the tool. Cap simulator calls in code, count verification runs separately and log every call outside the agent's reach.
4. Score with more than one metric. Report RMSE and R², plus a steady-state error and a dynamic index such as settling time, so that a saturated R² cannot hide an offset.
5. Run baselines. Include random search and a classical optimiser, ideally at both the agent's budget and a larger one, so you can separate reasoning from sample count.
6. Repeat and report spread. At least ten runs per method, with medians and ranges; Sonnet 5's wide dispersion on one test would be invisible in a single run [1].
7. Record time and cost. Wall-clock time, tokens and spend per run. TuneBench reports time only; you will need cost to make a deployment decision.
8. Keep a human sign-off. Treat the tuned parameters as a proposal that an engineer validates on held-out tests before they enter a study model.

The scoring functions are short. Note that R² here follows the paper's definition, with the target's mean in the denominator [1]:

import numpy as np


def rmse(target, sim):
    target, sim = np.asarray(target, float), np.asarray(sim, float)
    return float(np.sqrt(np.mean((target - sim) ** 2)))


def r2(target, sim):
    target, sim = np.asarray(target, float), np.asarray(sim, float)
    ss_res = np.sum((target - sim) ** 2)
    ss_tot = np.sum((target - target.mean()) ** 2)
    return float(1.0 - ss_res / ss_tot)


def settling_time(t, y, y0, y_final, band=0.02):
    """Time after which y stays within band * |step size| of y_final."""
    t, y = np.asarray(t, float), np.asarray(y, float)
    tol = band * abs(y_final - y0)
    outside = np.nonzero(np.abs(y - y_final) > tol)[0]
    if outside.size == 0:
        return float(t[0])
    if outside[-1] + 1 >= t.size:
        return float("nan")  # never settles inside the window
    return float(t[outside[-1] + 1])

Checklist

• Targets are perturbed from published defaults.
• The simulation or evaluation budget is enforced by the tool, not the prompt.
• Audit logs are written where the agent cannot edit them.
• At least two error metrics, plus one physically meaningful index, per test.
• Classical baselines at equal and larger budgets.
• Ten or more runs per method, reporting medians and spread.
• Runtime, tokens and cost logged per run.
• Results validated on tests the agent never saw.

Common mistakes

• Calling R² "accuracy". It is a fit statistic, and on a large step response it can exceed 0.99 while a steady-state offset remains.
• Quoting the best model as "AI agents". Only two of five agents met the bar in TuneBench; the smallest failed outright [1].
• Comparing budgets without saying so. Agents had 15 simulations and baselines 150. That favours the baselines on sample count, but it says nothing about wall-clock time or cost.
• Generalising from two cases. Two devices on an infinite bus is a strong start for a benchmark, not evidence for whole-network studies.
• Prompt-only guardrails. Budgets, protected files and permissions belong in code and configuration.

What to do this week

1. Read the TuneBench paper's results section and the repository's README to see the exact prompts and limits [1][3].
2. Pick one calibration or tuning task your team does by hand and write down its target tests and the metric an engineer would accept.
3. Wrap your simulator or test harness in a tool with a hard call cap and an audit log kept outside the agent's working directory.
4. Run a random-search baseline on the same task first, so any agent result has something to beat.
5. Add RMSE, steady-state error and settling time to whatever score you already report.

TuneBench is a useful signal that the strongest current models can do real engineering calibration when the problem is well posed and the tools are constrained. The work that makes that safe in production is the harness around the model. If you are designing agents that act on engineering systems, our AI agents page describes how we approach budgets, permissions and audit trails.

References

1. TuneBench: Benchmarking AI Agents for Power System Dynamic Model Calibration, Pomarico, Tessitore, Giannuzzi and Berizzi, arXiv:2611.04640v1, 3 November 2026.
2. ANDES: Python toolkit for power system modelling and simulation, CURENT, GitHub.
3. PowerAgentBench, TuneBench benchmark folder (dynamic/level3), GitHub.