Identify correlated failure domains, hidden shared dependencies, control-plane risks, and blast-radius boundaries in resilient system design.
A failure domain is a set of components that can become unavailable or incorrect because of the same event. A blast radius is the amount of the service, data, or organization affected when that event occurs.
Redundancy improves resilience only when copies fail independently enough for the scenario being protected. Two application instances on the same host protect against a process crash but not a host failure. Two clusters in one region may protect against a zone failure but not a region-wide identity, control-plane, or network problem.
| Layer | Example shared failure | Isolation response |
|---|---|---|
| Process | memory leak, crash, deadlock | multiple processes, restart, supervision |
| Host or node | hardware, kernel, local network | placement across hosts, rescheduling |
| Rack or power domain | switch, power distribution, cooling | spread across infrastructure groups |
| Zone or data center | facility, network, localized disaster | multi-zone placement and data strategy |
| Region or metropolitan area | regional service or connectivity disruption | geographically separate recovery or active service |
| Provider or platform | global control plane, shared software defect | reduced dependency, alternative operating path, sometimes another platform |
| Account, subscription, or project | deletion, policy error, quota, compromise | separate recovery boundary and delegated administration |
| Identity and key system | authentication outage, credential compromise, key loss | emergency access, separate trust path, key recovery |
| Deployment pipeline | bad artifact or configuration released everywhere | staged rollout, independent versions, rollback |
| Data model | logical corruption propagated across copies | point-in-time history, validation, isolated recovery copy |
| Organization | operator error, missing authority, unavailable staff | separation of duties, documented roles, exercises |
Geography is only one type of isolation. Administrative, temporal, software, and organizational independence can be equally important.
Independent failure is an assumption that must be justified. Replicas may share:
If a shared dependency fails, every visible replica can fail together. Draw dependencies from the user flow backward rather than looking only at the component you are trying to make redundant.
For each business-critical flow:
This approach often finds a small dependency that dominates the whole design. A multi-region application may still require one regional secret store. A redundant API may call one external payment endpoint. A secondary environment may depend on an artifact registry hosted only in the failed region.
Many platforms separate operations that configure resources from operations that serve live traffic.
A recovery design that depends on creating many resources during an incident inherits the availability of the control plane, quotas, identity, and capacity allocation. Pre-provisioned or continuously running capacity may reduce that dependency, but costs more.
This does not mean every control-plane operation is unsafe. It means the RTO must include the real operations and their failure modes. If a rapid recovery target depends on an API that may be impaired during the same event, test or redesign that dependency.
| Pattern | How it limits impact | Trade-off |
|---|---|---|
| Cell or stamp architecture | partitions users or workloads into independently operated units | more deployment and routing complexity |
| Separate accounts or projects | isolates policy, quota, credentials, and accidental deletion | cross-boundary governance and observability work |
| Staged deployment | limits a bad release to a canary or subset first | slower rollout and more release tooling |
| Bulkhead | prevents resource exhaustion in one pool from consuming every pool | lower utilization or duplicated capacity |
| Queue and backpressure | decouples failure and controls overload propagation | delayed work and operational queue management |
| Circuit breaker | stops repeated calls to an impaired dependency | requires safe fallback or explicit failure behavior |
| Read-only or degraded mode | preserves critical service without all dependencies | careful correctness and reconciliation design |
The goal is not maximum fragmentation. Every boundary adds cost, operational work, and potential configuration drift. Create a boundary where the reduced impact justifies the complexity.
Consider a replicated database:
| Protected failure | Potential design |
|---|---|
| process failure | local restart or another process |
| host failure | replica on another host |
| zone failure | replica or managed deployment across zones |
| region failure | cross-region replica or recovery from remote copies |
| bad write or deletion | point-in-time history independent of live replicas |
| account compromise | protected copy in a separate administrative boundary |
| software defect | version diversity, rollback, or recovery copy created before the defect |
No single topology covers every row automatically.
| Error | Better rule |
|---|---|
| “Two instances means no single point of failure.” | Check placement, traffic, data, identity, and control dependencies. |
| “Different regions are fully independent.” | Investigate global services, credentials, pipelines, DNS, software, and external dependencies. |
| “A managed service handles all resilience.” | Understand the service’s protected scope and the customer’s configuration responsibilities. |
| “Smaller blast radius always wins.” | Isolation has cost; align boundaries with business impact and operating capability. |
| “Only infrastructure fails.” | Releases, data, policies, credentials, and human decisions create correlated failures too. |
Continue with Active-Active and Active-Passive to decide how traffic and capacity should use those failure domains.
Further reading: Google Cloud infrastructure outage recovery architecture and Microsoft Azure availability zone and region strategies.