Record the location of the last file parsed. Use InputStream.skipSkip when parsing the log for the second time to avoid repeated analysis of data
A thread parses the log and stores the parsed data in the memory, and an additional thread is dedicated to processing the data (in this way, the parsing speed will be very fast, and 5K data should be in seconds)
There are already many mature products for log analysis, such as Logstash, which do not have to be stored in the database. Moreover, logging into the database also involves server IO performance. When the frequency of logging into the database is high, it will affect normal business.
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
Record the location of the last file parsed. Use
InputStream.skipSkip when parsing the log for the second time to avoid repeated analysis of dataA thread parses the log and stores the parsed data in the memory, and an additional thread is dedicated to processing the data (in this way, the parsing speed will be very fast, and 5K data should be in seconds)
There are already many mature products for log analysis, such as Logstash, which do not have to be stored in the database. Moreover, logging into the database also involves server IO performance. When the frequency of logging into the database is high, it will affect normal business.