Distinguish availability, reliability, durability, recoverability, fault tolerance, and resilience when evaluating a system design.
A system can excel at one resilience quality and fail badly at another. Architecture discussions become unreliable when available, durable, recoverable, fault tolerant, and resilient are treated as synonyms.
The most useful starting point is to ask what outcome the user or business actually needs.
| Quality | Practical meaning | Useful evidence |
|---|---|---|
| Availability | The service can successfully handle the required work now. | successful-request rate, usable time, user-visible error rate |
| Reliability | The system performs its intended function consistently over a stated interval and conditions. | SLO attainment, failure rate, correctness over time |
| Durability | Committed data remains intact and retrievable over time. | verified replicas, integrity checks, retained copies, loss history |
| Recoverability | Service and data can be restored to an acceptable state after disruption. | measured restore time, recoverable point, successful exercises |
| Fault tolerance | The system continues operating through a defined failure without depending on a long restore process. | automatic isolation, redundancy, transparent failover |
| Resilience | The system can absorb disruption, adapt, recover, and improve. | controlled degradation, recovery evidence, learning after incidents |
Availability is often measured from the user’s point of view. For request-driven systems, successful requests may be more informative than a simple server-up/server-down clock. A service that returns errors for one region or one customer group is partially unavailable even if every process is technically running.
Durability concerns the continued existence and integrity of data. A storage service may offer strong durability while the application using it is unavailable. Conversely, an application can remain available by serving cached or degraded responses while its authoritative data store is impaired.
Recoverability is demonstrated, not assumed. The existence of a backup file does not prove that the organization can locate it, decrypt it, restore it, replay required logs, reconnect dependencies, and return a valid service within the promised time.
An in-memory service accepts writes and responds successfully, but a power loss destroys the only copy. It was available before the failure, but committed data was not durable.
Several protected copies of a database exist, but the application cannot authenticate to the database after an identity outage. The data remains durable, but the service is unavailable.
Two database replicas continuously copy every change. An application bug deletes critical records, and the deletion is replicated immediately. Redundancy protects against a node failure, but without version history or point-in-time recovery it does not protect against logical corruption.
A nightly backup is tested and can restore the application in six hours. The system is recoverable, but it cannot continue through an outage with little interruption.
Compute runs across several zones, but DNS, identity, or a message broker remains a shared single point of failure. The visible compute topology looks redundant while the end-to-end user flow is not.
Users do not consume isolated components. They complete flows such as signing in, placing an order, processing a payment, or reading a report. The effective availability of the flow depends on every required component and the way those components fail.
For a simple serial dependency chain, adding individually reliable components can still reduce end-to-end availability because every required dependency must work. The answer is not to multiply service-level agreement percentages blindly. Instead:
A resilient system does not always preserve full functionality. During an incident it may:
That is different from silently returning incorrect data. A degraded mode needs an explicit contract: which operations continue, what users are told, how data consistency is protected, and how normal service returns.
| Layer | Availability question | Durability or recovery question |
|---|---|---|
| Traffic | Can healthy capacity receive requests? | Can routing be restored or changed safely? |
| Compute | Is redundant capacity running? | Can instances be recreated from trusted artifacts? |
| Application | Can critical flows degrade safely? | Are code and configuration reproducible? |
| Data | Can reads and writes survive the failure? | Can corruption, deletion, or site loss be recovered? |
| Identity and keys | Can systems and operators authenticate? | Are emergency access and key recovery available? |
| Operations | Will the problem be detected and handled? | Can the team execute and validate the recovery plan? |
| Error | Better interpretation |
|---|---|
| “There are three replicas, so the data is backed up.” | Replicas support availability; independent recovery points protect against replicated mistakes and corruption. |
| “The provider offers high availability, so the application is highly available.” | The application must use the feature correctly and protect every required dependency. |
| “The process is running, so the service is available.” | Measure successful user outcomes, not process existence. |
| “The backup job succeeded, so recovery is proven.” | Only a validated restore proves the recovery path. |
| “Active-active means zero downtime.” | Detection, routing, state, capacity, and client behavior still determine interruption. |
When evaluating a design, ask:
Continue with RTO, RPO, and Recovery Objectives to turn these qualities into measurable recovery requirements.
Further reading: Google SRE availability guidance and Google SRE data integrity.