concurrency - Java concurrently processes a large amount of data in a table, how to set checkpoints so that they can be used for failure recovery instead of repeatedly processing the processed data?

Question

Java concurrently processes a large amount of data in a table. How to set checkpoints for failure recovery instead of repeatedly processing the processed data?

For example, during the processing process, the system goes down and before kill -9 is forced to kill, set a checkpoint and resume processing from the checkpoint next time?

Answer 1

1. Write checkpoints regularly, so you are not afraid of downtime.
2. Data processing must be idempotent and deal with data that has been processed but has not been written to checkpoints.

If you use transactions to write checkpoints in real time, then the above problem will not happen.