The stretched cache: why the analogy with the database is false

Three ways to stretch a cache

A cache gets stretched in one of three ways:

1. one cluster across two sites, nodes split evenly;
2. master in one site, replica in the other;
3. an application in both sites talking to one cache.

The argument under all three is the same: our database is stretched with replicas and it works, so the cache can work that way too.

It is not a foolish argument. It is simply wrong, and wrong in three separate places.

A cache and a database do different jobs

The database is the source of truth. It must not be lost, and for that it is worth paying the travel time between sites.

A cache exists so that getting the same data again costs less than going to the source. "Less" does not necessarily mean "faster": it can equally be load taken off the source, an expensive recomputation not performed, a quota on an external call not spent, a hot key shielded. Either way the property is comparative: move the cache further away than the source and it is gone.

What follows measures one of those sides — latency — on the case least favourable to a cache.

Roughly 450 times slower than the database in your own site — that is the ratio the run prints.

The third way dies here. Half the requests will travel to the other site, pay the trip there and back, and return later than a trip to the local database would have.

It does not follow from the first two rows that a cache is pointless: what was measured there is a single row read by key from a small in-memory table — the cheapest thing a database does. What does follow is a rule rather than a number: going to the cache has to cost less than the source operation it replaces, at this distance. For a cheap read by key the answer at forty milliseconds is no. For an aggregate that takes a hundred milliseconds it is no longer no — and that case was not measured here.

Losing cache entries is normal; losing database rows is an incident

The second form: master here, replica there. The master answers the client OK without waiting for the replica. That is not a setting but declared behaviour: Redis uses by default asynchronous replication.

Now promote the replica in the master's place — that is, do the thing the replica is kept for.

Both rows have to be read together. What is lost is not "almost everything" and not "nothing" — it is whatever did not manage to reach the replica. Not one of the lost writes reported an error: the client got OK for all of them.

The loss window is set by the lag, not by the round trip

The experiment invites a rule: "at risk is everything written during the last round trip". The rule circulates in exactly that form, and it is wrong — here the round trip coincided with the loss by accident. This is checked by changing the write rate alone, over the very same link.

The link and the round trip are the same in all three. What changes is the lag, and with it the loss. So the rule in its correct form: at risk is whatever lies between the master's master_repl_offset and the replica's offset at the moment of the break — Redis reports both in INFO replication.

And this is where the analogy breaks. For a database that lost tail is an incident: nobody will recompute the row. For a cache, losing an entry is routine: miss, go to the source, write back.

So the machinery a stretched database is tolerated for is not needed by default by a cache. And it would still have to be paid for: WAIT, the command usually proposed to "make replication synchronous", makes a write 171.6 times more expensive — 41.73 ms instead of 0.24 ms on every write.

What that price is made of cannot be told from a single point — and the guess "two round trips" suggests itself. The same experiment at three distances answers more precisely, and answers something else.

The ratio to the round trip is one at all three distances. So WAIT costs exactly one round trip to the replica, and grows with distance one for one.

This block is published in its second revision, and the first one was wrong — wrong in its numbers rather than its conclusion. It gave ratios of 5.20, 2.10 and 1.55, with "a distance-independent term of 42–44 ms of unknown origin" left over. The origin turned out not to be Redis at all: the relay standing in for the inter-site link had no TCP_NODELAY, and the pair of Nagle's algorithm and delayed acknowledgement was adding forty milliseconds to every operation where several small messages crossed the link in succession. The instrument was measuring itself. The full account is in the detailed version of this article.

And WAIT does not buy a guarantee either: it reports how many replicas acknowledged, but it cannot cancel the write — on the master it has already been applied and is already visible to readers. For a cheaply rebuilt cache, paying a round trip on every write is hard to justify. But if warming takes hours, or the source will not survive a cold-start stampede, replication durability acquires a value — and it has to be compared against the cost of refilling, not against zero.

"Stretch" means different things for a database and a cache

An asynchronous hot standby of Postgres in another site is a separate system: read only, lagging, and the application knows it. It decides nothing on the master's behalf.

The caveat is mandatory: "the database is stretched" is not a description of an architecture. Postgres also has synchronous replication, in which a commit waits for the remote standby, and there are databases that use a quorum. What has to be compared is one specific replication model against another, with both ends named — below it is an asynchronous Postgres replica against a Redis cluster.

A Redis cluster across two sites is one system that makes decisions by vote. The survival condition is in the specification, and it has two requirements:

Everyone remembers the first: you need a majority. The second gets skimmed. It is the second that does all the damage.

The main result: both halves failed

Six nodes, three in each site, the masters split two and one. The diagram is symmetric, and that is what convinces.

The map before the link was cut:

The cache was gone everywhere. In neither of the two data centres.

Site 2 is straightforward: one master out of three, no majority.

But site 1 did have the majority — two masters of three. It had the votes. There was simply nobody to promote: the replica of the master left in site 2 stayed in site 2 as well (look at the map above). Nobody covered that master's slots, and a cluster with incomplete coverage (cluster-require-full-coverage, yes by default) stops accepting writes.

Why the reads failed too

Reads are governed by a different setting — cluster-allow-reads-when-down, no by default. This invites the conclusion that the observed behaviour is "wider than the documented one"; the conclusion is wrong — the behaviour is documented. The chain is: slot coverage incomplete → the cluster is marked as failed → a node of a failed cluster does not serve reads either. Checked with the same layout, run twice.

A SET on the same node came back CLUSTERDOWN in both runs. With yes the node answers only for those keys whose slots stayed with it: out of twenty, the surviving half returned fourteen and redirected the rest to the master behind the severed link. The setting does not bring the cache back — it turns "nothing works" into "some reads work on part of the key space".

Hence the rule: a majority of votes is necessary but not sufficient. The full condition has three parts: a majority of masters on your side, plus a reachable replica of every lost master, plus that replica being fresh enough — one lagging beyond cluster-replica-validity-factor will not start an election. Freshness did not get in the way in these runs, and the run prints the line that shows it.

You cannot check this by eye on a diagram: a diagram shows how many nodes there are, not whose replica sits where. The cluster arranges the replicas itself — at creation and after every failover.

You can move a replica by hand, and then one half will survive. But only one: what you are choosing that way is which data centre survives the break.

The boundary of that conclusion has to be drawn precisely. The claim is not "both data centres cannot work"; it is this: in one ordinary Redis cluster you cannot get a symmetric partition in which both halves independently keep accepting writes and are then merged back automatically. Other outcomes are possible — two independent caches, read-only operation in a reduced form, a product with active-active and an explicit conflict resolution model — it is just that they are no longer "one stretched cluster", and none of them was measured here.

What to do instead

• A separate cache in each site, filled by its own application from its own source. A link failure is not an event for it.
• In an ideal steady state the price is one fill of every hot key in every region. That is a lower bound, not an estimate: "two misses and that is it" would be a deception, the real load comes from eviction, restarts, failovers, deployments, invalidation, cold starts and traffic moved between sites. Hence something a single shared cache did not have — two independent stampede domains.
• Divergence between sites is acceptable, but a TTL alone does not manage it. The observed divergence is the sum of four terms: the lag of the source itself, the invalidation delay, the freshness policy (the TTL proper and its relatives — early refresh, sliding expiry, serving a stale value while it is recomputed) and the aftermath of failures and recovery. What deserves the name "budget" is the sum.
• Synchronous cross-site invalidation is an expensive deal: waiting for confirmation brings back the round trip we walked away from, and puts it in the hot path of a write. If waiting really is required, the problem is not the TTL: "acknowledge the write only once another region can see it" is a requirement on the consistency model of the cache layer, met by versioning, by routing the reader back to the same region, or by taking the cache out of the correctness path.
• A disposable cache must be recoverable from an authoritative source or from a repeatable computation. That source may be a database, an object store, an event log, an external API or a canonical computation — what matters is not which, but that the path back is known.
• A separate cache per site removes the inter-site link from the path to the cache, but does not make the application independent of it. If writes still go to a database master in the other site, a break is still an event for the application.
• A shared cache is still a cache. The circulating rule goes: "if the cache has to be shared, then what is in it is not a cache". That is wrong: a shared external cache serving several applications is a normal arrangement — it takes load off the source better than a set of in-process caches and survives a deployment without starting cold. The red flag is not "shared" but irrecoverability. The test: delete the cache entirely. It got slower and the data came back — that is a cache. Sessions, locks and counters were lost — that is a state store, and a different conversation.

When a stretched cache is nonetheless defensible

• If one designated site has to survive the break, not both. This works — but you are picking the winner in advance, there will be no cache in the second site, and the replica layout has to be re-checked after every failover.
• If a replica in the other site is there not to serve traffic but to avoid filling the cache from scratch. The terms: replicas do not promote themselves, you will be switching over by hand, and whatever lay between the master's and the replica's offsets at the moment of the break will be lost. For a cache the last part is tolerable — those entries get recomputed on a miss.
• If the write rate is sparse. The second row of the failover run and the offsets run: what is dangerous is not the volume of data but the lag the write rate manages to create. Check it from the difference of offsets, not from the distance between sites.
• If the source operation costs more than the round trip. The latency run compared a cache in the other site against the cheapest thing a database can do. For an aggregate that takes a hundred milliseconds the comparison is a different one, and a cache at forty milliseconds wins it. No numbers for expensive operations were taken here.

What all four have in common is what is absent: not one of them promises that both halves of one cluster independently keep accepting writes.

Three ways to stretch a cache

A cache gets stretched in one of three ways, and they are so different that all they share is the word "stretch".

1. One cluster across two sites, nodes split evenly. The most convincing version on a whiteboard: three boxes on the left, three on the right, a break down the middle — and it looks as though each half will carry on.
2. Master in one site, replica in the other. A direct tracing of the database: there is a replica next door there, let there be one here.
3. An application in both sites talking to one cache. Formally this is not even a stretched cluster — just one address serving two sites.

They have to be taken apart separately, because they break in different places. But all three grow from one thought, and that is where to start.

The analogy everything grows from

The argument behind stretching is usually left unsaid, because it feels self-evident: our database is stretched across two data centres with replicas, it has worked for years, nobody complains — so the cache can work that way too.

This is not a foolish argument. It rests on real and successful experience: a stretched database does work, and the engineer who built it has earned the right to generalise. The generalisation simply turns out to be wrong — and not in a detail, but in all three places where it leans on the words "the same way".

A database and a cache have different jobs. They have different costs for losing one record. And the word "stretch" means different things for them. What follows is one difference per section, and each one has a measurement.

A caveat: what it was measured on, and what carries over

Every number in this article was taken on Redis 7.0.15, and what follows holds for Valkey and for current Redis as well. Here is what that means and what it does not.

Valkey is a fork of Redis, and says so itself: Valkey is a fork of the open-source Redis (REmote DIctionary Server) database created in 2009 by the Italian hacker Salvatore "antirez" Sanfilippo. That licenses carrying over not the numbers, but the contract: where the Valkey documentation describes the same behaviour in the same words, the article's claim rests on a quotation from it rather than on an analogy. That is the case for asynchronous replication and for the cluster's survival condition — both wordings in the fork match the Redis ones down to the product name, and both are quoted below.

Appealing to a shared past is not enough, though: behaviour changes with versions, and that has to be checked rather than assumed. So every substantive claim in this article was re-run on Redis 8.10.1 and Valkey 9.1.2 — the table is in a section of its own below. None of them diverged.

The numbers carry over nowhere. They are not needed as absolute values here anyway: what carries meaning is the ratio, the sign, and which half of the cluster answered and which did not.

Difference one: a cache and a database have different jobs

The database is the source of truth. It must not be lost, and for that it is worth paying a round trip between sites: a row that took forty milliseconds to reach the second site is still cheaper than a row that exists nowhere.

A cache exists so that getting the same data again costs less than going to the source. "Less" does not necessarily mean "faster": it can equally be load taken off the source, an expensive recomputation not performed, a quota on an external call not spent, a spike smoothed out, a hot key shielded. Either way the property is comparative, not absolute, and so it can be lost without breaking a single setting — simply by moving the cache further away than the source.

What follows measures one of those sides — latency — and measures it on the case least favourable to a cache.

The run states the ratio outright: a cache in the other site is roughly 450 times slower than the database in your own.

The third way — "an application in both sites talking to one cache" — breaks right here, though not in general: it breaks for this class of queries. Half the requests will travel to the other site, pay the round trip and come back later than a trip to the local database would have. If the cache was there for latency on cheap reads, it does not reduce that latency, it increases it, and on top of that it puts one more failure-prone system between the application and its data.

What this measurement does not say, and that matters more than the numbers. The first two rows nearly coincide, and it does not follow that a cache is pointless. What was measured is the cheapest operation a database performs: five thousand rows, everything in memory, a read by primary key, no contention for connections. A cache is not deployed against that query — it is deployed against connections, aggregates over millions of rows, and the same query repeated a thousand times a second.

Exactly one thing follows, and it is a rule rather than a number: a cache's advantage is taken from a comparison with the source, so the question to check is not "is Redis fast" but "does going to the cache cost less than the source operation it replaces, at this distance". For a cheap read by key out of a warm table, the answer at forty milliseconds is no. For an aggregate that takes a hundred milliseconds it is no longer no — and that case was not measured in this article.

Difference two: losing a row and losing a cache entry cost different things

The second way — master in one site, replica in the other — traces the database literally. The first thing the measurement shows is that the distance is visible.

The lag by itself is not a problem. The client got its OK long before: the master answers without waiting for the replica, and that is not a quirk of configuration but a declared contract.

The same page in Valkey says the same thing: Valkey uses by default asynchronous replication — and word for word from there on.

One misunderstanding is worth clearing away here, because half the arguments rest on it: asynchrony is not what separates a cache from a database. Postgres streaming replication is asynchronous by default too, and the documentation names the consequence outright:

So both have a loss window. What differs is not the existence of the window but the cost of falling into it — and that is where this is going.

The lag becomes a problem at the moment the replica is promoted in the master's place — that is, in the very scenario the replica is kept in the other site for.

Both rows have to be read at once, or the first one is carried away as "almost everything is lost" and the second as "nothing is". Neither is true: what is lost is whatever did not manage to reach the replica. Not one of those writes reported an error: the client got OK for all of them.

What sets the loss window, and why it is not the round trip

The experiment invites a rule: "at risk is everything written during the last round trip before the break". The rule circulates in exactly that form, and it is wrong: here the round trip coincided with the loss by accident — two hundred fast writes happened to fit inside it. This is checked by changing the write RATE alone, over the very same link.

The link is the same in all three, and so is the round trip. What changes is the lag, and with it the loss.

Hence the rule in its correct form: at risk is whatever lies between the master's master_repl_offset and the replica's offset at the moment of the break. Redis reports both numbers in INFO replication, and their difference shows how much you would lose if the failover happened right now. The round trip only sets a lower bound on how quickly the lag can come down to zero; the lag itself also depends on the write rate, the size of the commands and on whether the replica keeps up in applying them.

This is where the analogy with the database breaks a second time. The loss window, as we have just seen, exists for the database too. What differs is not the mechanics but the cost: for a database that lost tail is an incident — nobody will recompute the row, it exists nowhere. For a cache, losing an entry is the norm the whole thing is built around: miss, go to the source, write back. So the machinery that justifies a stretched database is not needed by default by a cache — and would still have to be paid for.

The bill can be read directly. WAIT is the very instrument usually proposed to "make replication synchronous".

Forty-one point seven milliseconds instead of nought point two four is the cost of waiting, added to every write. What that cost is made of cannot be told from a single point — and the guess "that is two round trips" suggests itself. The same experiment at three distances answers more precisely, and answers something else.

The last column is one at all three distances. So WAIT costs exactly one round trip to the replica, and grows with distance one for one: the round trip grew by 30 ms and the price grew by 30.62 ms; the round trip grew by 40 ms and the price grew by 39.87 ms. Less than a millisecond and a half remains above the round trip, and that is the work of Redis itself rather than distance.

This block is published in its second revision, and the first one was wrong — wrong in its numbers rather than its conclusion. It gave ratios to the round trip of 5.20 / 2.10 / 1.55, concluded that no fixed number of round trips described the price, and left a term of 42–44 ms above the round trip whose cause the measurement does not establish. The conclusion "with distance the price grows by one round trip" was right even then, because it rested on the differences. The term, however, turned out to be a defect of the measuring instrument rather than a property of Redis — and that is the next section.

The property bought is weaker than expected as well: WAIT reports how many replicas acknowledged, but it does not cancel the write if fewer did. By the time it answers, the write on the master has been applied and is already visible to readers — there is nothing to roll back with. The documentation says so outright:

So that 171.6-fold price buys not a guarantee but a report on the number of copies. Whether it is worth it is a question not about Redis but about what refilling the cache costs; the article returns to it in the section on what to do instead.

Checking the instrument: what the inter-site link actually was

This section came out of an external review, and the question it asked sat one level below all the previous ones. The earlier ones asked what we concluded from a measurement. This one asked what the measurement actually modelled.

In every run the link between data centres is portrayed not by tc netem but by a relay in user code: a process listens on a port, reads a chunk, waits, and passes it on. The objection was specific: the delay is applied to every chunk recv() returns, and TCP preserves no message boundaries. So what was being measured was not "Redis plus a fixed propagation delay" but "Redis plus a delay multiplied by however many chunks the stream broke into".

Cross-checking against netem, as the review asked, is impossible in this environment: the kernel is built without CONFIG_NET_SCH_NETEM, and tc qdisc add ... netem answers Error: Specified qdisc kind is unknown. Privileges have nothing to do with it — the queuing discipline simply is not in the kernel. So the cross-check had to be against what was available: measuring the instrument itself.

The first thing it showed was that the suspicion was aimed at the wrong thing.

There is the origin of the "42–44 ms term" the WAIT block found and could not explain. It belongs entirely to the measuring instrument. Redis sets TCP_NODELAY on its own connections; the relay creates two new connections, and the flag was not set on those. From there the familiar pair takes over: Nagle's algorithm holds a small chunk back until the previous one is acknowledged, and the delayed acknowledgement is held for up to forty milliseconds. Hence the value of just over forty, its independence from distance, and the fact that it only appeared where several small messages travelled the link in succession.

Now the review's original objection, also in numbers. How many times the delay is applied per operation:

Three chunks per operation: the write itself, the acknowledgement request, and the acknowledgement. But only two of them are on the critical path — the first two travel in the same direction and could have left together. With a serializing relay they do not: while the thread sleeps it does not read the next chunk. The cost of that is measured directly:

The review was right: the serializing link costs exactly one extra one-way flight at every distance. Not some arbitrary amount — precisely the delay it applies to a chunk.

And here is the part more instructive than the error itself. While the forty- millisecond stall sat in both links, this difference was invisible: it drowned in an artefact four times its size. Two defects of the rig were masking each other. Finding either one by staring at the resulting milliseconds was impossible — only taking the instrument apart did it.

Last comes the calibration, which is also the boundary of what the rig is good for:

How to read the rest of the article after this. Where one chunk travels in each direction per operation — the latency run and the replication-visibility run — the relay yields exactly the round trip, and the numbers can be taken as they are. Where several messages travel in succession — replication under load, command pipelining, WAIT — the instrument was adding something of its own, and both terms have now been measured and removed. A third kind of error remains and cannot be removed here: the relay has no queues, no loss, no jitter and no bandwidth limit. It is an approximation of delay, not a packet-level network emulator, and absolute milliseconds on streaming operations should be read that way.

The qualitative conclusions — sign, ratio, the order of events, which half of the cluster answered — do not depend on any of this.

Difference three: "stretch" means different things

That leaves the main one — the first way, "nodes split evenly". And here the analogy breaks not in the pricing but in the meaning of the word.

Take the configuration people usually mean when they say "the database is stretched": an asynchronous hot standby of Postgres in another site. It is a separate system: read only, lagging, and the application knows it. It does not vote and does not decide the fate of the first one; when the link goes down the master keeps working and the replica keeps lagging until the link comes back.

The caveat here is mandatory, or the section commits the very mistake the article argues against. "The database is stretched" is not a description of an architecture: Postgres also has synchronous replication, in which a commit waits for the remote standby (if the standby is the last one in a synchronous group, setting this to on will result in commits waiting for the standby to confirm receipt), and there are databases that use a quorum. What has to be compared is not "a database" against "a cache" but one specific replication model against another specific replication model — with answers about synchrony, about who decides on a failover, and about behaviour under a partition. What is compared below is an asynchronous Postgres replica against a Redis cluster, and both ends are named for a reason.

A Redis cluster across two sites is one system, making decisions by majority vote. The condition under which it survives a break is written into the specification, and it contains two requirements, not one:

Everyone remembers the first requirement — "you need a majority". The second one gets skimmed, and this article is essentially about it. Valkey words it the same way: Valkey Cluster is able to survive partitions where the majority of the primary nodes are reachable and there is at least one reachable replica for every primary node that is no longer reachable.

Before getting to the even split, it is worth looking at the layout that seems the most cautious: all masters in one site, all replicas in the other.

Neither half changed roles during the break, and on both clusters every replica had master_link_status: up before the link went down — so nothing was blocked by staleness. That detail matters two sections down.

Site 1 works, site 2 answers nothing at all. The replicas did not become masters: promotion needs the vote of a majority of masters — Once the replica receives ACKs from the majority of masters, it wins the election — and every master stayed on the far side of the break.

That is the first result worth taking away: the second half is not a spare. If the first one is lost, it will not replace it — not on its own, at any rate.

The culmination: both halves failed

Now the layout the whole idea is built around. Six nodes, three in each site, the masters split two and one. On a whiteboard it looks symmetric, and it is the symmetry that convinces.

The map the run printed before cutting the link, on the cluster where site 1 was the half under observation:

The cache was gone everywhere. In neither of the two sites, for no key at all.

Before taking apart why, it is worth saying how this result was obtained, because the method was corrected after the review. Previously both halves were checked in sequence on one cluster: stop the far one, poll the near one, resume, stop the near one, poll the far one. That cannot be done. After the first check the topology need not return to its original shape: a replica may have been promoted, the configEpoch may have changed, slots may have moved — and the second half would then be measured on a different cluster. Two checks in sequence are not two sides of one partition. Now each side is observed on its own fresh cluster with the same, hand-specified topology.

Why site 2 lost is the expected part. It holds one master out of three, it has no majority, and the specification leaves no room: Redis Cluster is not available in the minority side of the partition. That is how it should be: if a minority kept serving writes, the cluster would drift apart into two diverging data sets.

Site 1 is the interesting one. It holds the majority of masters — two out of three. It has enough votes to promote a replica. And it answers CLUSTERDOWN all the same.

The reason is visible on the map above, and it is worth tracing with a finger. The master left in site 2 is node 35445. Its replica — node 38349 — stayed in site 2 as well. So there is nobody to promote in site 1: that master's slots are covered by no reachable node. This is precisely the second requirement from the specification, the one that gets skimmed: a reachable majority is not enough, you also need a reachable replica of every lost master.

Then the setting that turns the failure total rather than partial kicks in:

This invites the conclusion that the observed behaviour is "wider than the documented one": the documentation talks about writes, while a GET also came back CLUSTERDOWN. The conclusion is wrong. Reads are governed by a different setting, and that one is documented too — cluster-allow-reads-when-down, no by default. The chain is this: slot coverage incomplete → the cluster is marked as failed → a node of a failed cluster does not serve reads either.

This is checked with the same layout, run twice.

A SET on the same node came back CLUSTERDOWN in both runs.

And here is what actually follows from it — more interesting than the corrected mistake. With yes the node answers only for those keys whose slots stayed with it: out of twenty, the surviving half returned fourteen, and redirected the rest to the master that is now behind the severed link. Writes are refused either way.

So the setting does not bring the cache back. It turns "nothing works" into "some reads work on part of the key space", and a third of the keys — the slots of the master behind the break — are unreachable whatever its value.

The rule the measurement was set up for. A majority of votes is necessary but not sufficient. The full condition has three parts, and the third is usually not stated at all:

a majority of masters on your side
  + a reachable replica of every lost master
  + that replica fresh enough to be allowed to stand

The third part is not a quibble: a replica's freshness is governed by cluster-replica-validity-factor, and a replica lagging beyond the allowance will not start an election. It did not get in the way in the runs above — every replica had master_link_status: up before the break, which the run prints — but the specification's condition is incomplete without it, and relying on two parts out of three means checking the wrong thing.

Under an even split the condition does not always hold — and, more to the point, you cannot check it by eye on a diagram of three boxes: a diagram shows how many nodes there are, not whose replica sits where. The cluster assigns replicas itself, at creation and after every subsequent failover.

And this is where the central conclusion comes from. Stretching is done for symmetry — "so that both survive". Symmetry is exactly what kills it: in an evenly split cluster the survival condition holds for nobody, and at the moment it matters the cache turns out to be missing in both sites at once.

The layout that survives, and what it actually chooses

The condition can be satisfied by hand: keep in site 1 not its own replica, but the replica of the master that is going to site 2.

The map after the rearrangement — one replica moved, nothing else:

Now site 1 survives: it had the votes and it had someone to promote. The run states it outright — node 36267: replica of 53509 -> master. The replica brought home took over the lost master's slots, and all three nodes came back to state ok.

But symmetry did not appear. There is still no cache in site 2, and that is not a defect of the layout but its point. What a layout like this chooses is which data centre survives the break.

The boundary of that conclusion is worth drawing precisely, or it turns into a slogan. The claim is not "both data centres cannot work"; it is this: in one ordinary Redis cluster you cannot get a symmetric partition in which both halves independently keep accepting writes and are then merged back automatically. The reason is not the layout but the fact that the cluster is one and the decision it takes is one.

Other outcomes are possible — it is just that they are no longer "one stretched cluster": two independent caches, read-only operation in a reduced form, a product with active-active and an explicit conflict resolution model (Conflict resolution is handled by conflict-free replicated data types (CRDTs)), conflict resolution on the application side. Each has its own price, and this article measured none of them.

The fragility of the arrangement is visible in passing. Between the two runs there is a single replica swap. It is written down nowhere as a requirement, is not checked at startup, and does not survive an automatic failover: after the first master failure the roles are redistributed, and the "correct" layout silently becomes the one that failed in both halves.

The same claims on current builds

Every number above was taken on Redis 7.0.15. That branch is old: by September 2026 the current Redis Open Source line is 8.10.x and Valkey has moved to 9.1. The cluster implementation changed noticeably between Redis 7 and 8, and "it matched once" does not make a property invariant. So the same claims were re-run on all three builds — not performance, but the claims themselves.

Not one claim in this article diverged: the master waits for the replica nowhere, WAIT costs a round trip everywhere, the half holding the majority of masters fails entirely everywhere, and cluster-allow-reads-when-down decides the fate of reads everywhere. This also closes the question of Valkey: the article used to rest on its documentation and say honestly that nothing had been measured on the fork. Now it has been.

What this table does not do. It does not carry the numbers of the earlier runs over to the new versions, and it does not replace them. The published baseline stays on 7.0.15; the table answers a different question — whether the behaviour has drifted.

What to do instead

The analysis is useless without this section, so here it is by points. The first answer is a good default, not the only one: the cases where people choose otherwise are listed at the end of the section.

A separate, independent cache in each site. Not a replica, not half a cluster — its own installation, filled by its own application from its own source. A link failure is then not an event for the cache: both sites carry on, each with its own data.

The caveat matters, and without it this becomes a slogan. Independent regional caches remove the WAN dependency from the path to the cache, but they do not make the application as a whole independent of the inter-site link. If writes still go to a database master in the other site, a break is still an event for the application — it has merely stopped being an event for the cache. Those two availabilities have to be separated explicitly, or "an independent cache" turns into a promise it never made.

The price is extra misses. The first request for a key misses in site 1 and goes to the source; the first identical request in site 2 misses again. But this has to be stated carefully: in an ideal steady state, with no eviction and no restarts, the minimum extra cost is one fill of every hot key in every region. That is a lower bound, not an estimate.

The real load on the source is set not by the arithmetic of keys but by eviction, restarts, failovers, resharding, deployments, key version changes, invalidation, cold starts and traffic being moved between sites. Every one of those events refills the cache, and now it does so in each data centre separately. Hence something a single shared cache did not have: two independent stampede domains, two different hit rates and two sets of graphs to watch.

Divergence between sites is acceptable and is managed — but not by a TTL alone. Two independent caches will inevitably diverge, and the question is not "how do we avoid this" but "how many seconds may we diverge by". A TTL is the most direct knob here, which is exactly why it is easy to mistake for the whole answer — "a TTL just is the budget for inconsistency, named in seconds". That is wrong, and here is what the observed divergence is actually made of:

• the lag of the source itself. If each site has its own read replica of the database, its lag sits underneath the divergence of the caches rather than in place of it;
• the invalidation delay — how long the "this key changed" event takes to reach the second site, if it travels at all;
• the freshness policy — the TTL proper, and with it its relatives: early refresh, sliding expiry, serving a stale value while it is recomputed;
• the aftermath of failures and recovery — after a failover or a traffic move, the cache in one site is cold while the other is warm.

A TTL is one term out of four. What deserves the name "budget" is the sum, and it has to be worked out from your own topology rather than from this article.

Synchronous cross-site invalidation is an expensive deal, and a TTL is not what settles it. The temptation is there: we deleted a key here, delete it there too, and wait for confirmation. That brings back the round trip we have just walked away from, and brings it back into the hot path of a write. For a disposable cache the deal is almost always bad: locality was the whole reason for splitting the cache by region in the first place.

But if waiting really is required, the conclusion "then the TTL was chosen wrong" is too narrow, and wrong by more than a nuance. A requirement that a write be acknowledged only once another region can see it is a requirement on the consistency model of the cache layer, not on the lifetime of a key. A TTL will not deliver it: you need a different model (versioning, routing a reader back to the same region, bounded staleness with an explicit bound), or the cache has to come out of the correctness path. A TTL governs how far you may diverge, not whether you may diverge at all.

And WAIT is not meaningless for a cache — it just rarely pays off. The blunt form — "WAIT is meaningless for a cache" — is an error of the same kind as the previous two. For a cheaply rebuilt cache, paying a round trip on every write really is hard to justify: losing an entry is by design. But caches differ. If warming takes hours, if the source will not survive a cold-start stampede, if behind it sits a rate-limited external API or a heavy recomputation — then replication durability acquires a value, and it has to be compared not against zero but against the cost of refilling and the acceptable loss window. An industry account of the same trap puts it without hedging: a cache can be a big and highly damaging blast radius. Note that this very article later defends a remote replica kept for a warm cache: same argument, different instrument.

The source of truth need not be a database, but there has to be one. The circulating form — "anything that cannot be recomputed from the database has no business being in the cache" — is too narrow for a systems discussion. The rule is wider: a disposable cache must be recoverable from an authoritative source or from a repeatable computation. That source may be a database, an object store, an event log, an external API, another service, or a canonical computation result. Which one does not matter; what matters is that it exists and that the path back is known.

A shared cache is still a cache, and "shared" is not evidence. The circulating rule goes: "if the cache has to be shared, then what is in it is not a cache". That is not so. A shared external cache that several applications talk to is a normal and common arrangement: it takes load off the source better than a set of in-process caches, it removes divergence between application nodes, and it survives a deployment of the application without starting cold.

The red flag is not the word "shared" but irrecoverability. A cache stops being a cache when losing an entry turns not into a miss but into a loss of state: the only copy of a session, a lock, a counter, process state, data that exists nowhere else. The test for this is simple and needs no argument about terminology: delete the cache entirely and see what happened. It got slower and the data came back — that is a cache, and there is no reason to stretch it across sites. Money, sessions and locks were lost — that is not a cache but a state store, and the conversation about it is a different one; it needs an article of its own, and it will get one.

Six layouts side by side, and what they cost

Everything above takes apart three ways to stretch a cache and one way not to. There are more, and holding them in your head as a list is awkward — so here they are together. The table is not a measurement: it contains no numbers, and cannot, because three of its six rows were never measured by anyone. It is a map for deciding what to measure next.

Read this table not as a ranking but as a list of questions to ask about your own topology. In this order: what exactly is cached and can it be lost; what does the source operation the cache replaces cost; how far away is the source; how much divergence is acceptable; what has to survive a break; how long does the cache take to warm again. The answers pick the row — not how the arrangement looks on a diagram.

When a stretched cache is nonetheless defensible

This article argues against a stretched cache, not against every case of one. There are four in which a stretched configuration is defensible — and all four rest on the same measurements.

When one designated site has to survive the break, not both. The layout in the last run works: site 1 answered on all three nodes. If you have a primary site and a standby rather than two equals, "the cluster survives in the primary" is a legitimate goal. But on three conditions: you must understand that you are picking the winner in advance; that there will be no cache in the standby site; and that the replica layout has to be re-checked after every failover, because the cluster arranges it itself.

When a replica in the other site is there not to serve traffic but to avoid filling the cache from scratch. A large warm cache is hours of work and a noticeable load on the source at a cold start — which is what the warning that a service is likely to be unable to handle the resulting load on its dependencies is about. A replica promoted by hand after an incident removes that load. The terms are honest and both were measured: the replicas do not promote themselves (three nodes in state fail, zero masters) and a promotion loses whatever did not arrive — at risk is the lag at the moment of the break. For a cache the second is acceptable: the lost entries get recomputed on a miss. Which is exactly why this holds only for a cache, and only if you accept switching over by hand.

When the write rate is sparse. The second row of the failover run is not a caveat but a separate result: after one second of silence all two hundred writes were on the replica, none lost. The offsets run shows the same thing again and more precisely: what is dangerous is not the volume of data but the lag the write rate manages to create. If the replica keeps up on average, a promotion has almost nothing to lose — and you can check that from the difference of offsets instead of guessing.

When the source operation costs more than the round trip. The latency run compares a cache in the other site against the cheapest thing a database can do: a read by primary key out of a warm table. For an aggregate that takes a hundred milliseconds the comparison is a different one, and a cache at forty milliseconds wins it. The rule from that measurement is stated without numbers and works both ways: going to the cache has to cost less than the source operation it replaces, at this distance. No numbers for expensive operations were taken here and quoting any would be dishonest; but neither does this measurement give any right to forbid such a cache.

What all four have in common is what is absent from them. Not one of them promises that both halves of one cluster independently keep accepting writes. As soon as the word "both" appears in the requirement, nothing on this list will do, and the choice has to come from elsewhere: a separate cache in each site, a layered cache, or a product with active-active and an explicit conflict model.

How to reproduce the numbers

Eight scripts, all against a live Redis, all printing what the processes returned.

python3 bench/stretched-cache/replication.py   # master and replica across sites
python3 bench/stretched-cache/cluster.py       # three cluster layouts
python3 bench/stretched-cache/latency.py       # cache in the other site vs local database
python3 bench/stretched-cache/readsdown.py     # reads in a failed cluster
python3 bench/stretched-cache/offsets.py       # the loss window and the offsets
python3 bench/stretched-cache/waitcost.py      # the price of WAIT at three distances
python3 bench/stretched-cache/relaycheck.py    # the measuring rig itself
python3 bench/stretched-cache/versions.py      # the same claims on current builds

redis-server is needed on the PATH, plus redis-cli for the cluster run and a running PostgreSQL for the latency one (address in DE_TEST_DATABASE_URL). A different build can be pointed at through REDIS_SERVER_BIN and REDIS_CLI_BIN — that is how the version table was taken. The scripts leave nothing behind: nodes come up in temporary directories and are killed in finally.

The limits of the rig, stated outright

The delay is produced by a relay in user code, not by netem. The relay adds delay to every chunk recv() returns, and TCP preserves no message boundaries. This is a reproducible approximation of latency, not a packet-level network emulator: it has no queues, no loss, no jitter and no bandwidth limit. Plain request/response measurements can be read directly — the calibration block shows the relay yields exactly the round trip there. Streaming replication and WAIT would ideally be confirmed separately through netem or network namespaces; in the environment this article was measured in, the kernel is built without CONFIG_NET_SCH_NETEM and there was nothing to do it with. Instead the instrument itself was measured, and two of its defects were found and removed.

The break in the cluster run is portrayed with STOP. The process stays alive and the port stays open, but the node neither answers nor sends heartbeats: to the remaining half it is indistinguishable from a node behind a severed link. Killing them was not an option — the second half had to be polled too. Each side is observed on its own fresh cluster with a hand-specified topology, so the aftermath of the first check cannot reach the second; a real network partition through firewall rules or network namespaces would be more accurate still.

The relay has a cut() switch, and it is not decorative: a break between data centres is not "it got slow" but "it stopped going through at all", and portraying it as increased latency would be wrong.

The version boundary

The core properties are checked against the current Redis and Valkey documentation. The numbers of the main run were taken on Redis 7.0.15 and kept as a reproducible baseline. Redis 8.x developed the cluster implementation considerably, so the key claims — replication, WAIT, the even split, cluster-allow-reads-when-down — are additionally re-run on Redis 8.10.1 and Valkey 9.1.2, and are not treated as invariant merely because they matched once.

The published run: Redis 7.0.15, PostgreSQL 16.13, one host, link 20 ms one way — except for the runs where the distance is the variable of the experiment. Twenty milliseconds were chosen not to make things look worse but as an ordinary distance between data centres within one country.

Three of these scripts appeared after the first review of this article: they check exactly the three places where a conclusion had been drawn from a single point — reads in a failed cluster, the loss window, and the price of WAIT. Two more appeared after the second review, and they check not Redis but the measurement. Four of those five checks found an error, and all four corrections stand in the text where they belong rather than as an afterword.

What measured this

The numbers in this article come from these scripts. Each one opens from here, together with the record of the run: what it was measured on, what came out, and how to read it.

• bench/stretched-cache/replication.py
• bench/stretched-cache/cluster.py
• bench/stretched-cache/latency.py
• bench/stretched-cache/readsdown.py
• bench/stretched-cache/offsets.py
• bench/stretched-cache/waitcost.py
• bench/stretched-cache/relaycheck.py
• bench/stretched-cache/versions.py
• bench/stretched-cache/link.py