Beyond Unit Tests: Testing How Your System Fails and Recovers

A green unit test suite tells you each function does what you meant. It says nothing about what happens when the database is slow, a retry storm starts, or a whole zone disappears. This guide sets out the tests and safeguards that fill that gap, with commands and configuration you can try this week.

What unit tests cannot tell you

Google's SRE team puts it plainly in their twenty-year retrospective: unit tests "have intentionally limited scope, and are super helpful, but they also don't fully replicate the runtime environment and productionized demands that might exist" [1]. Their answer is integration testing, including checking that jobs and tasks can perform a cold start and that components work together as a system [1].

The failures that cause real outages are usually interactions: a timeout that is longer than the caller's timeout, retries that multiply load on a struggling backend, a cache that every request suddenly misses. Google's chapter on cascading failures shows how these spread, and recommends limiting retries per request, considering a server-wide retry budget and always using randomised exponential backoff [2]. None of those properties exist inside a single function, so no unit test can check them.

Inject faults before production does

The cheapest place to learn how a dependency failure spreads is a test environment where you control the network. Toxiproxy, an open-source TCP proxy from Shopify, sits between your service and a dependency and lets you add latency, cut connections or take the dependency down on demand [8]. Point the service at the proxy, then script the faults:

# Route the service's database connection through Toxiproxy
toxiproxy-cli create -l 0.0.0.0:15432 -u payments-db:5432 payments_db

# 1. Slow dependency: add 1500 ms (+/- 250 ms) to responses
toxiproxy-cli toxic add -t latency -a latency=1500 -a jitter=250 payments_db
#    ...run the suite: assert on timeouts, fallbacks and error rates...
toxiproxy-cli toxic remove -n latency_downstream payments_db

# 2. Black hole: stop all data and close connections after 5 s
toxiproxy-cli toxic add -t timeout -a timeout=5000 payments_db
toxiproxy-cli toxic remove -n timeout_downstream payments_db

# 3. Hard down: disable the proxy, then restore everything
curl -s -X POST localhost:8474/proxies/payments_db -d '{"enabled": false}'
curl -s -X POST localhost:8474/reset

Toxics are directional; a latency toxic applies to the server-to-client stream unless you add one for upstream too [8]. The HTTP API on port 8474 is unauthenticated by default, so never expose it to an untrusted network [8].

For cloud resources, AWS Fault Injection Service (FIS) runs experiments built from actions, targets and stop conditions, and can stop or roll back an experiment automatically. AWS strongly recommends planning experiments and running them in pre-production before production [9].

Make every dependency fail safely

Fault injection is only useful if the code has something to prove. Three safeguards matter most: a timeout on every remote call, bounded retries with randomised backoff [2], and a circuit breaker that stops calling a dependency that is clearly failing.

Netflix popularised this pattern. Adrian Cockcroft described wrapping "a backend circuit breaker pattern" around each dependency using their Hystrix library, and using Latency Monkey to inject failures and high latency so that "timeouts and circuit breakers [are] calibrated properly" [4]. Hystrix has since moved to maintenance mode, and Netflix recommends active projects such as Resilience4j for new work [5].

In Resilience4j a breaker moves from CLOSED to OPEN when the failure rate, or the share of slow calls, reaches a threshold, but only once a minimum number of calls has been recorded. While OPEN it rejects calls; after a wait it moves to HALF_OPEN and lets a small number of trial calls through [6]. A Spring Boot configuration looks like this [7]:

resilience4j.circuitbreaker:
  instances:
    paymentsDb:
      registerHealthIndicator: true
      slidingWindowType: COUNT_BASED
      slidingWindowSize: 50
      minimumNumberOfCalls: 20
      failureRateThreshold: 50
      slowCallDurationThreshold: 2s
      slowCallRateThreshold: 50
      waitDurationInOpenState: 30s
      permittedNumberOfCallsInHalfOpenState: 5

resilience4j.retry:
  instances:
    paymentsDb:
      maxAttempts: 3
      waitDuration: 200ms
      enableExponentialBackoff: true
      exponentialBackoffMultiplier: 2
      enableRandomizedWait: true
      randomizedWaitFactor: 0.5

resilience4j.timelimiter:
  instances:
    paymentsDb:
      timeoutDuration: 2s

The numbers are starting points, not recommendations. The point of the Toxiproxy run is to check them: with 1,500 ms of added latency, does the breaker open, does the fallback serve, and does the error rate stay inside your SLO?

Rehearse disaster recovery, not just resilience

Google's lesson eight separates two kinds of testing. Resilience testing "verifies that your service or system could survive in the event of faults, latency, or disruptions"; recovery testing "verifies that your service can transition back to homeostasis after a full shutdown" [1]. Both belong in business continuity planning, and the authors also suggest tabletop sessions that walk through "what if" scenarios [1].

Lesson two explains why rehearsal matters: an outage "is a terrible opportunity to try a risky load-shedding process for the first time". Practise recovery mechanisms beforehand so you know they work and that people know how to run them [1]. Restoring a backup you have never restored, or failing over to a Region you have never served from, is a guess, not a plan.

Automate mitigation, sized to the risk

Google reports that automating mitigations reduces mean time to resolution (MTTR): "If there's a clear signal that a particular failure is occurring, then why can't that mitigation be kicked off in an automated way?" [1]. The same article warns that "the riskiness of a mitigation should scale with the severity of the outage", after a risky load-shedding step during a YouTube incident caused a cascading failure instead of fixing one [1]. It also recommends a "Big Red Button": a simple, pre-identified action that reverts a risky change [1].

A sensible rule: automate containment that is reversible and well tested, and keep a human approval step for actions such as moving write traffic between Regions until you have rehearsed them.

Measure whether users noticed

A test passes or fails on user impact, not on whether a pod restarted. The SRE Workbook recommends defining a service level indicator (SLI) as good events divided by total events, such as successful HTTP requests over all requests. The error budget is 100% minus the SLO: with a 99.9% SLO and 3 million requests over four weeks, you can afford 3,000 errors [3]. Record how much budget each fault injection run consumed. If one experiment spends a quarter of the month's budget, you have found something worth fixing.

A playbook for one critical journey

1. Choose a journey and its SLI. For example "payment submitted and confirmed", measured as successful confirmations over attempts [3].
2. List its dependencies and mark each as one you must have or one you can degrade without.
3. Add a cold-start integration test that boots the service with real configuration and calls the journey end to end [1].
4. Put each dependency behind a proxy in the test environment and script latency, timeout and down scenarios [8].
5. Write the hypothesis first: "With 1,500 ms on the database, the breaker opens within 30 seconds and the error rate stays under 0.1%."
6. Run it in CI or a nightly job and fail the build when the hypothesis fails.
7. Rehearse recovery from a full shutdown in pre-production, then with FIS and stop conditions in more realistic environments [9].
8. Automate the mitigation you just proved, and give it a Big Red Button [1].

Checklist

• Every remote call has an explicit timeout shorter than its caller's.
• Retries are capped per request and use randomised exponential backoff [2].
• Each critical dependency has a circuit breaker and a defined fallback.
• At least one integration test starts the service from cold.
• Latency, black-hole and hard-down faults are scripted and repeatable.
• Recovery from a full shutdown has been rehearsed within the last quarter.
• Every test reports SLI and error budget consumption.
• Automated mitigations are reversible, logged and have a manual override.

Common mistakes to avoid

• Mocking the failure away. A mock returns an error instantly; a real slow dependency holds a thread for seconds. Test with real latency.
• Retrying at every layer. Attempts multiply across layers, so three layers of three attempts can turn one user request into 27 calls to a backend that is already struggling [2].
• Breakers that never trip. If the minimum number of calls is higher than the traffic your test generates, the breaker cannot open [6].
• Testing recovery once. Systems change; a recovery path that worked a year ago may not work now.
• Automating the riskiest action first. Start with alerting and containment, as the table above suggests [1].
• Using Hystrix for new code. It is in maintenance mode [5].

What to do this week

1. Pick your most important user journey and write down its SLI as a good/total ratio.
2. Grep your service for HTTP and database clients without an explicit timeout, and fix the worst one.
3. Run Toxiproxy in front of one dependency locally and add 1,500 ms of latency. Note what breaks.
4. Count how many layers retry the same call, and cap them.
5. Book a one-hour tabletop session: "what if our primary database Region is unavailable?"

Reliability comes from testing how the system fails and recovers, and unit tests can't cover that. If you would like a second pair of hands designing fault injection, recovery drills and on-call automation, our SysOps 24/7 team can help.

References

1. Lessons Learned from Twenty Years of Site Reliability Engineering, Google SRE.
2. Addressing Cascading Failures, Site Reliability Engineering (Google).
3. Implementing SLOs, The Site Reliability Workbook (Google).
4. Interview: Adrian Cockcroft on High Availability, Best Practices, and Lessons Learned in the Cloud, InfoQ, October 2013.
5. Hystrix README: Hystrix Status, Netflix on GitHub.
6. CircuitBreaker, Resilience4j documentation.
7. Getting Started (Spring Boot configuration), Resilience4j documentation.
8. Toxiproxy, Shopify on GitHub.
9. What is AWS Fault Injection Service?, AWS FIS User Guide.