Data analysis - java data mining analysis ideas

Question

Let me talk about my idea first: monitor the prices of many items on Taobao at the same time, and then obtain the price trend (I plan to write a crawler to obtain this myself, and then store it in the database), but a problem arises

How should I store data so that I can analyze which items have the largest price fluctuations in the last 2 or 7 days (preferably any number of days), and how to implement this better. I can't say that I have to traverse the database every day to find the price difference. Can an experienced master give me some advice?

Answer 1

I can’t say that I have to traverse the database every day to find the price difference, right? Do you think the cost is too high? Then put the hot data in Cache and persist it to the database after this hot time.

Hotspot data can be but is not limited to:

• Data just collected in the past few days

• The most commonly analyzed date range, such as 2 days, 7 days, 1 month

Answer 2

This has nothing to do with java, right?

Answer 3

Data analysis methods are not strongly related to Java. From a certain perspective, Java is just a tool that can transfer your ideas to calculations and enable you to communicate with computers. There is a haoop ecosystem based on Java, which can Get to know

Answer 4

Fast changing data that needs to be analyzed --> Hot data --> redis, stored in memory;
Long-term historical data --> Cold data --> mysql / oracle, stored on disk.