Sometimes it feels like we understand MySQL, and we understand Redis, yet in interviews we never answer well and are constantly stumped. Where is the problem?
The answer is: what the interviewer is testing is not a single-area skill, but the ability to apply several technologies in combination.
Take the interviewer’s classic torture-test question — how do you guarantee MySQL and Redis cache consistency in a concurrent scenario? — as an example.
For business logic that reads a lot, writes little and demands high performance, we usually add a Redis cache layer between the application server and the MySQL database, to improve query efficiency, relieve the pressure on MySQL, and avoid a performance bottleneck in MySQL.
With this problem, if after the data is stored we are only in a read-only scenario, no MySQL/Redis cache consistency problem arises at all, so what really needs consideration is the data consistency problem in a concurrent read/write scenario.
If we do not analyse it and simply answer the question of how to guarantee MySQL and Redis cache consistency in a concurrent scenario using only our knowledge of MySQL and Redis, it is very hard to answer well, because what looks like a very simple scheme is in fact riddled with holes.
A Simple Scheme Riddled with Holes
Let’s first look at the problems that arise with the simple update-database, delete-cache, and update-cache schemes.
Update the Cache, Then Update the Database
The conclusion first: not considered.
The reason is that after the cache update succeeds, the database update may fail, leaving the database with the old value and the cache with the new value. As a result, for all subsequent read requests, as long as the cache has not expired or been correctly rewritten, the data stays completely inconsistent the whole time! And the current value in the database is the old one, while the correctness of business data should be judged by the database.
So if the cache update succeeds but the database update may fail, can we just re-update the cache?
Setting aside the performance problems caused by having to re-query data from a single table or multiple tables when re-updating the cache and then update the data again, there may also be data changes during that period that plunge you into dirty data again. In fact, concurrency consistency problems still occur.
As long as the cache has been updated, subsequent read requests — before the database update, and before the database update fails and the cache re-update is about to happen — will basically all hit the cache, and the data returned at that moment is all dirty data that has not been persisted.
Update the Database, Then Update the Cache
Not considered.
The reason is that when the database update succeeds but the cache update fails, the database has the latest value while the cache has the old value. As a result, for all subsequent read requests, as long as the cache has not expired or been correctly rewritten, the data stays completely inconsistent the whole time!
Even when both the database update and the cache update succeed, this scheme still has consistency problems caused by concurrency, as shown in the figure below (click the image to view it full size):
You can see the data consistency problems that exist in concurrent multi-write multi-read scenarios.
Delete the Cache First, Then Update the Database
Not considered, but it can be considered after using a delayed double-delete strategy.
Adopting the “delete the cache first, then update the database“ scheme is a common way to try to solve this problem.
This approach is fairly simple in logic, easy to understand and implement, and in theory after the old cache is deleted, the next read fetches the latest data from the database.
But in the extreme concurrent case, after the cache deletion succeeds, if a large number of concurrent requests come in, they will go straight to the database, putting enormous pressure on it. And this scheme can still suffer from data consistency problems.
From the figure above we find that after the cache is deleted, if concurrent read request 1.1 comes in, the cache lookup definitely misses, so it reads the database; but because at this point the operation 2. update database x=10 has not yet completed, what is read is still the old value x=5, and after setting the cache, once operation 2. update database completes, the database holds the new value 10 while the cache holds the old value, causing a data inconsistency problem.
For this we can first do a small optimisation, namely the delayed double-delete strategy. That is, after updating the database, first wait a while (the wait time is roughly the response time of that read request plus a few tens of milliseconds), and then delete the cache. The purpose is to ensure that the read request has finished (it has already read the old data at 1.2 read database, and will update the cache later within that request), so the write request can delete the dirty cache data caused by the read request, guaranteeing that all read requests after the second cache deletion read the latest value.
It can be seen that the key point of this optimisation is how long to wait before deleting the cache again, but that time is judged from the response times of historical query requests and will fluctuate in practice. This also means that if the delay is too short, data inconsistency still occurs; if the delay is too long, the period during which data is inconsistent becomes longer.
In addition, the delayed double-delete strategy also has to consider what to do if deleting the cache again fails.
Because a failed deletion will cause all subsequent read requests, as long as the cache has not expired or been correctly rewritten, to stay completely inconsistent the whole time! This will be discussed further in the technical optimisation scheme below.
Update the Database First, Then Delete the Cache
Relatively recommended.
The adopted “update the database first, then delete the cache” strategy is basically the same as the small delayed double-delete strategy optimisation we made for “delete the cache first, then update the database”, and it still needs to consider how to handle a failed cache deletion.
Comparing “update the database first, then delete the cache” with “delete the cache first, then update the database” purely on their own merits, in most cases “update the database first, then delete the cache” is regarded as the better choice, for the following reasons:
Data consistency: this method leans more towards maintaining eventual consistency of the data; even if the cache deletion fails, it guarantees that data consistency will not be damaged in the long term.
User experience: with “delete the cache first, then update the database”, if the database update fails, users may keep seeing old data until the cache expires. By contrast, “update the database first, then delete the cache” can avoid this to some degree.
But this scheme likewise has data consistency problems, as shown in the figure below.
After the database data is updated, the cache is deleted too. Next, read request 3.1 and write request 3.2 come in at the same time.
The read request first reads the cache, finds a cache miss, so it queries the database; and just as it is about to update the cache, write request 3.2 has already finished updating the data and deleting the cache, and only afterwards does read request 3.1 update the cache. In the end the value in the database is the new value while the value in the cache is the old value.
The Optimised Scheme
From the simple schemes above, none of them seems to truly solve the problem of MySQL data versus Redis cache data consistency in a concurrent scenario.
One thing to explain here: if the business requires strong consistency, then no matter how you optimise the cache strategy it cannot be satisfied, and the best approach is not to use a cache at all.
Strong consistency: it requires that whatever the system writes is what is read back; the user experience is good, but implementing it usually has a large impact on system performance.
The solution is read-write serialisation, and this scheme would greatly reduce the system’s processing efficiency and drastically lower throughput.
Also, in large distributed systems, distributed transactions are in fact mostly not used, because the maintenance cost is too high and the complexity is high as well. So in distributed systems we generally advocate eventual consistency: this consistency level constrains the system such that after a write succeeds it does not promise that the written value can be read immediately, nor does it promise how soon the data will become consistent, but it will ensure as far as possible that after some time level (say, seconds) the data reaches a consistent state.
Now let’s continue optimising…
Delayed Double-Delete Strategy + Retry Mechanism
From the “delete the cache first, then update the database” scheme in the riddled-with-holes simple schemes above, we can see that the delayed double-delete strategy is arguably a fusion of “delete the cache first, then update the database” and “update the database first, then delete the cache”, and can solve most data consistency problems in business logic handling.
But we still left one unresolved question earlier: what to do if deleting the cache again fails?
——-Of course, the remedy is to go on deleting this cache key, and the remedy method is retry.
The retry mechanism can launch a new coroutine in the current process (a user-space lightweight thread in Golang) to retry; it can also be put into a message queue to retry; or it can first launch a new coroutine to retry 3 times, and after the retries fail, put it into a message queue to retry. The figure below shows retrying via a message queue.
When retrying in a new coroutine, note that you use the new context context.Background(), not the context of the current request.
Generally message queues support highly reliable queues, such as RabbitMQ, Kafka and so on. These message queues provide very strong message delivery, asynchronous processing and persistence features, and can effectively solve data synchronisation problems.
This scheme still has some needs, such as: choosing a suitable delay wait time before deleting the cache; the number of retries and the interval between them when retrying the cache deletion in the coroutine; whether the message queue needs to retry after a cache deletion failure, and so on.
Reading the binlog to Delete the Cache Asynchronously
The retry-on-cache-deletion mechanism is not bad, it is just that it intrudes into a lot of business code.
In fact, you can also optimise it like this:
Use Canal to capture the binlog and send it to an MQ queue to evict keys asynchronously.
The application that deletes the cache confirms the handling of this update message through the ACK manual mechanism, deletes the cache, and guarantees data cache consistency.
Evicting keys asynchronously is simpler than waiting to compare and update cached data, because a single piece of cached data may involve querying, aggregating and sorting data from multiple tables.
Although this scheme does not look bad either, it introduces extra components (such as Canal and a message queue) which add a fair amount of complexity: you need to maintain and monitor the running state of these components and keep them running normally.
Scheduled Tasks
In certain business scenarios, Redis and MySQL data can also be synchronised by means of a scheduled task.
The specific approach is to read data from Redis on a schedule, compare it with the data in MySQL, and synchronise if the data in Redis has changed.
Although this method is fairly simple to implement, you need to watch the timeliness of the synchronisation: if the time interval is set improperly, it may cause the synchronised data to be lost or inaccurate.
Dual-Write Consistency
Update the cache/delete the cache at the same time as updating the database — the so-called “dual write“.
This ensures that after the database is updated, the data in the cache is also the latest, thereby reducing the window during which data is inconsistent.
Concurrency control: in high-concurrency scenarios, when multiple requests update the same piece of data at the same time, without proper concurrency control there may be data inconsistency problems. So here we introduce a distributed lock and transactional operations:
Using a distributed lock: before performing the dual write, acquire a distributed lock (such as Zookeeper, Redis’s SETNX command, and so on) to ensure that only one thread/process can perform the dual write at a time.
Transaction handling: for cache systems that support transactions (such as Redis’s MULTI/EXEC command) and MySQL transactions, you can put the Redis cache and MySQL update operations into a transaction to ensure that either everything succeeds or everything fails.
Of course, in a “dual write” strategy, besides concurrency control, you can combine it with the retry and scheduled strategies mentioned above in order to cope with data inconsistency problems in extreme cases.
In addition, you can add an alerting mechanism to the failure-handling logic so that developers and operations staff are notified in time.
Reposted from: 皇子谈技术
