In order to avoid repeated calculations when performing some time-consuming queries, distributed lock services can be used.
Only one operation is performed at the same time, and similar operations are waiting to be retried.
The following code (fetch_with_dist_lock ) defines a fetcher and an updater.
If the fetcher cannot obtain the data, the updater is used to update. After the update is successful, the result is returned through the fetcher.
There are also some situations where we only want to update a certain data, and there are multiple updaters. But the update operation is not atomic. Then we will do it through update_with_dist_lock.
def fetch_with_dist_lock(mc_store, mutex_key, fetcher, updater, lock_time=3*60*1000, sleep_time=100, retry_times=3): i = 0 while i < retry_times: i += 1 need_update, results = fetcher() if need_update: if(mc_store.add(mutex_key, lock_time)): try: updater() continue finally: #release the distribute mutex anyway mc_store.delete(mutex_key) else: time.sleep((sleep_time*1.0)/1000) continue return results #too much tries, but failed still. return None def f_wrapper(f, *args, **kwargs): def _(): return f(*args, **kwargs) return _ def update_with_dist_lock(mc_store, mutex_key, updater, lock_time=60*1000, sleep_time=100, retry_times=5): i = 0 while i < retry_times: i += 1 if (mc_store.add(mutex_key, lock_time)): try: updater() return True finally: mc_store.delete(mutex_key) else: time.sleep((sleep_time*1.0)/1000) continue return False