Understand synchronous and asynchronous replication, quorum, lag, write ownership, conflict resolution, promotion, and the relationship between consistency and RPO.
Replication places copies of state in more than one failure domain. It can improve availability, read locality, and recovery speed, but every design must decide when a write is considered committed and what happens when replicas cannot communicate.
The central trade-off is not simply “more replicas are safer.” It is the relationship among latency, availability, consistency, write ownership, failure scope, and operational recovery.
| Model | Write acknowledgement | Strength | Trade-off |
|---|---|---|---|
| Synchronous | waits for the required replicas or quorum before success | can provide a very tight RPO inside the protected scope | adds latency; may reject writes when replicas cannot agree |
| Asynchronous | primary acknowledges before every remote copy confirms | lower write latency and practical over long distance | latest acknowledged writes may be absent after sudden promotion |
| Semi-synchronous or bounded | waits for part of the replica set or applies a lag bound | balances latency and protection | guarantees depend on exact acknowledgement and failure behavior |
Do not infer the model from the word “replicated.” Read the product or database guarantees: which replicas confirm, what consistency clients observe, and what state can be lost during failover.
Synchronous replication across nearby zones can often meet application latency requirements. Synchronous replication across distant regions adds network round-trip time to the write path and may reduce write availability during a partition.
Asynchronous cross-region replication is common because it preserves primary write performance, but it creates a nonzero exposure window. The effective RPO depends on actual lag, queue durability, failure detection, and promotion behavior, not only a configured target.
Ask these questions:
A system can return quickly and remain available by acknowledging from one location, but that choice changes what can survive an immediate failure.
| Topology | Write behavior | Recovery concern |
|---|---|---|
| Single writer with replicas | one authoritative writer; replicas copy state | safe promotion, lag, stale reads, fencing old writer |
| Multi-writer | several locations accept writes | conflicts, ordering, partition behavior, convergence |
| Partitioned ownership | each shard or tenant has one writer | ownership transfer, routing, hot partitions |
| Quorum-based | reads and writes consult a defined subset | quorum loss, latency, stale or conflicting replicas |
| Log shipping or change capture | ordered changes are transferred and replayed | missing segments, replay order, schema compatibility |
Topology labels hide implementation details. A single-writer database may expose readable replicas but require manual promotion. Another may automatically elect a new writer. A multi-writer system may provide strong consistency, eventual convergence, or application-visible conflicts.
Common observations include:
These are conceptual categories. Product guarantees vary. The application must tolerate the selected behavior. Inventory, account balances, access revocation, recommendation feeds, and telemetry may justify different models.
When several locations accept writes to the same logical item, the system needs a deterministic outcome.
| Strategy | Works well when | Risk |
|---|---|---|
| Last-writer-wins | updates are replaceable and clock/order semantics are acceptable | silently discards concurrent intent |
| Application merge | domain rules can combine changes correctly | complexity moves into application logic |
| Conflict-free data types | operations are designed to converge mathematically | limited data models and implementation complexity |
| Partitioned ownership | each key has one authoritative writer | ownership transfer and routing become critical |
| Synchronous consensus | strong ordering is required | latency and reduced availability during partition |
Conflict resolution is a business rule, not only a database setting. Merging two profile edits differs from reconciling two withdrawals from the same balance.
Before promoting a replica:
Automatic promotion reduces RTO but needs conservative safety rules. Promoting an incomplete replica or allowing the old writer to continue can convert an availability incident into an integrity incident.
Replication usually copies the current stream of changes, including:
Use independent historical recovery points for these failures. Replication and backup solve complementary problems.
| Data class | Possible priority |
|---|---|
| financial ledger | strict integrity, auditable ordering, minimal or zero acknowledged-write loss |
| shopping cart | read-your-writes and high availability; reconciliation may be possible |
| product catalog | read availability and geographic locality; bounded staleness may be acceptable |
| telemetry | high ingestion availability; limited loss or duplication may be acceptable if measurable |
| derived analytics | reproducibility from source may permit looser RPO |
| access revocation | stale authorization may be a security risk, requiring tight propagation |
One universal consistency choice is rarely optimal for every data set.
| Error | Better rule |
|---|---|
| “Asynchronous replication means the RPO equals the configured interval.” | Measure real lag and promotion behavior under load and failure. |
| “Synchronous replication guarantees all disaster recovery.” | It protects a defined scope but can replicate logical corruption and share administrative failures. |
| “Eventual consistency means random or incorrect data.” | It describes convergence timing; application correctness still requires explicit rules. |
| “More writers always improve availability.” | Multi-writer operation adds conflict and partition semantics that may threaten integrity. |
| “Failover is only a routing change.” | Writer authority, lag, fencing, retries, and reconciliation determine safety. |
Continue with Failover, Failback, and Recovery Testing to turn the architecture into a verified operational capability.
Further reading: Google Cloud infrastructure outage recovery architecture and Google SRE data integrity.