Agentic AI in banking: efficiency you can defend to a risk committee

Most banking AI agents that fail in production do not fail on a single bad answer. They fail across a conversation: the customer flunks an identity check in step one, and three turns later the agent cheerfully reads out a balance. This post shows how to get real efficiency from agents in banking and fintech by scoping their tools, testing whole sessions against mock systems, and measuring reliability rather than demo quality.

Where agents actually fit in a bank

An "agentic" system is one where a large language model (LLM) decides which steps to take and which tools to call, rather than following a fixed script. Anthropic draws a useful line here: workflows are systems where LLMs and tools are orchestrated through predefined code paths, while agents are systems where the model directs its own process and tool use [1]. The same guide recommends finding the simplest solution possible and only adding complexity when needed, and warns that autonomy brings higher costs and the potential for compounding errors [1].

That advice matters more in financial services than almost anywhere else. The Bank of England and FCA's 2024 survey of 118 UK firms found that 75% were already using AI, with a further 10% planning to within three years [2]. Of the reported use cases, 55% involved some degree of automated decision-making, but only 2% were fully autonomous; 24% were semi-autonomous, with humans overseeing critical or ambiguous decisions [2]. The same survey found 46% of firms reported only a partial understanding of the AI they use, largely because so much of it comes from third parties [2].

The third-party point is structural, not incidental. As Patrick McKenzie notes, community banks mostly do not have programmers on staff and rely on core processors such as Fiserv, Jack Henry and FIS [3]. Many banks gain new capability through fintech partnerships instead: Cross River Bank, for example, originates loans for fintechs that are then largely sold on to funding sources the fintech has arranged [3]. An agent in that world sits on top of APIs owned by somebody else. Your control surface is the tools you expose to it and the tests you run before it ships.

The failure that turn-level testing misses

Most tracing tools score an agent turn by turn: was this reply relevant, polite, grounded? Cekura, a YC-backed testing company, made the case against that approach in its March 2026 launch post with a banking example: the user fails verification in step one, but the agent hallucinates and proceeds anyway [4]. Every individual turn can look fine. The bug lives in how the turns relate to each other, so a turn-level evaluator marks the later steps green while a session-level judge fails the conversation [4].

Two other ideas from the same post are worth copying whatever tool you use. First, simulate the customer: generate realistic conversations, including awkward and adversarial ones, and run them before every release to catch regressions [4]. Second, mock the tools. Running simulations against real APIs is slow and flaky, so define tool schemas, behaviour and return values in a mock layer, and the simulation exercises tool selection and decision-making without touching production systems [4]. In a bank, the mock layer is also what lets you test "account frozen", "core timeout" and "customer is a vulnerable person" without waiting for those cases to happen.

Limit what the agent can do before you test what it does

OWASP's Top 10 for LLM applications calls the core risk here excessive agency: damaging actions taken in response to unexpected, ambiguous or manipulated model output, with three root causes of excessive functionality, excessive permissions and excessive autonomy [5]. Its mitigations translate almost directly into a banking design [5]:

• Minimise extensions and their functionality. A card-replacement agent needs order_replacement_card, not a generic "call any core API" tool.
• Minimise permissions and execute in the user's context. The agent should act with the authenticated customer's entitlements, never a service account that can see every account.
• Require approval for high-impact actions. Payments, limit changes and address changes go to a human or to a step-up authentication.
• Complete mediation. Enforce authorisation in the downstream system, not in the prompt. The prompt is advice; the API is the control.

Anthropic's guide makes the matching engineering point: put real effort into the agent-computer interface, meaning the tool definitions and their documentation, and test them thoroughly, because the model can only be as precise as the tools it is given [1].

A playbook for one production-grade banking agent

1. Pick one bounded journey. Card replacement, address change, dispute intake or document chasing for a loan file. Write down the start state, the end state and every rule that applies (identity, vulnerability flags, cut-off times).
2. Build it as a workflow first. If a fixed sequence of LLM calls and tool calls handles most cases, ship that [1]. Only let the model choose its own steps where the path genuinely varies.
3. Define narrow tools. One tool per business action, typed inputs, clear error messages, executed with the customer's permissions, authorisation checked in the API [5].
4. Build the mock layer. Mock every tool with schema-valid responses, including failures: timeouts, frozen accounts, partial data [4].
5. Write a scenario library. Happy paths, failed verification, customers who change their mind, prompt-injection text pasted into a free-text field, and every incident you have ever had.
6. Judge whole sessions. Start with deterministic rules (no protected tool before verification, no payment without approval), then add a model-based judge for tone and accuracy [4].
7. Run each scenario several times and track pass^k. One pass proves little; see below.
8. Shadow, then supervise. Run against live traffic without acting, then with human approval on high-impact steps, and only then widen autonomy for the steps that have earned it.
9. Feed production back into tests. Every escalation or complaint becomes a new scenario, so the suite grows with real behaviour.

Measure reliability, not a single lucky run

LLM agents are not deterministic, so a scenario that passes once may fail on the next run. The τ-bench paper (Yao et al., 2024) proposed pass^k: the probability that an agent succeeds on all of k independent attempts at the same task [6]. Its headline finding was sobering. Even strong function-calling agents completed fewer than 50% of tasks in its simulated retail and airline settings, and were inconsistent, with pass^8 below 25% in retail [6]. For a customer-facing bank process, pass^k is the number that matters, because every customer is another trial.

The session check and the metric are both short. This runs as-is in Python 3.8 or later:

from math import comb

PROTECTED = {"get_balance", "update_address", "make_transfer"}

def session_passes(trace):
    """Fail the whole session if a protected tool runs before identity is verified."""
    verified = False
    for step in trace:
        if step["tool"] == "verify_identity":
            verified = step["result"] == "pass"
        elif step["tool"] in PROTECTED and not verified:
            return False, f"turn {step['turn']}: {step['tool']} ran without verified identity"
    return True, "ok"

def pass_hat_k(successes, trials, k):
    """pass^k for one task (tau-bench): chance that k of the n trials, drawn at random, all succeed."""
    return comb(successes, k) / comb(trials, k)

trace = [
    {"turn": 1, "tool": "verify_identity", "result": "fail"},
    {"turn": 2, "tool": "get_balance", "result": "ok"},
    {"turn": 3, "tool": "update_address", "result": "ok"},
]
print(session_passes(trace))
# (False, 'turn 2: get_balance ran without verified identity')

# 7 passes out of 8 runs looks good until you ask for consistency
print(pass_hat_k(7, 8, 1), pass_hat_k(7, 8, 4), pass_hat_k(7, 8, 8))
# 0.875 0.5 0.0

Average pass^k across tasks to get a suite score. An agent with 87.5% single-run success on a task has only even odds of getting it right four times in a row, which is a very different story for a risk committee.

Computer-use agents and verifiable rewards

Many bank processes still run through screens with no API: loan origination systems, CRM, internal reporting. Agents that operate a browser or desktop, called computer-use agents, are one way in. Halluminate, in its August 2025 launch post, described training such agents in simulated websites using reinforcement learning with verifiable rewards (RLVR), where the agent is rewarded only when an automatic checker confirms the task was done correctly [7]. The company said one customer saw about a 20% improvement in date-picking on its simulated flight-booking site [7]; that is a vendor's own report, not an independent benchmark.

The idea transfers even if you never train a model. A verifier grounded in business rules (the loan record has the right fields, the right status, the right audit entry) is exactly the session judge you need for evaluation. If you cannot write that verifier, you are not ready to automate the screen.

Checklist before an agent touches customers

• The journey has a named business owner and written rules.
• Every tool is narrow, typed and authorised in the downstream system.
• High-impact actions need human approval or step-up authentication.
• A mock layer covers every tool, including failure responses.
• The scenario library includes failed verification and prompt injection.
• Sessions are scored as a whole; turn scores are secondary.
• pass^k is reported per scenario, with a release threshold agreed in advance.
• Full session traces are retained so a reviewer can replay any decision.
• Third-party model and platform dependencies are listed, with an exit plan.

Common mistakes

• Security in the prompt. "Never reveal balances before verification" is not a control. Enforce it in the API [5].
• Testing against production APIs. Slow, flaky and risky; mock them [4].
• Reporting one-run pass rates. They overstate reliability; use pass^k [6].
• Starting with an agent. A fixed workflow is cheaper and easier to audit when the path is known [1].
• Treating vendor results as evidence. Use them to choose what to test, then test it on your own scenarios.

What to do this week

1. Choose one journey and write its rules on a single page with the business owner.
2. List every tool your current prototype can call and delete or narrow any that the journey does not need.
3. Write ten scenarios, at least three of them failures (failed verification, core timeout, injected instructions).
4. Add the session rule above to your test harness and run each scenario eight times.
5. Report pass^1 and pass^8 side by side to whoever signs off the release.

Done this way, the efficiency is real: staff stop handling routine journeys, and risk teams can see why the agent is allowed to. If you want help scoping tools, building the mock layer or running evaluations for an agent like this, our AI agents team does this work with UK firms.

References

1. Building effective agents, Anthropic, 19 December 2024.
2. Artificial intelligence in UK financial services 2024, Bank of England and FCA, 21 November 2024.
3. Community banking and fintech, Patrick McKenzie, Bits about Money, 22 October 2021.
4. Launch HN: Cekura (YC F24), testing and monitoring for voice and chat AI agents, Hacker News, 3 March 2026.
5. LLM06:2025 Excessive Agency, OWASP Top 10 for LLM Applications.
6. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Yao, Shinn, Razavi and Narasimhan, arXiv:2406.12045, June 2024.
7. Launch HN: Halluminate (YC S25), simulating the internet to train computer use, Hacker News, 11 August 2025.