About building billion-level cmd5 database with mysql and millisecond-level query (complete process)

Foreword:

I have also been playing with databases recently, and I feel that ordinary machines are still a little behind when it comes to databases. Full-text search can take more than a minute if it is slow.

But the cmd5 library is very good, a billion-level database, millisecond level.

Okay, let’s start. First, you need a mysql database.

Environment:

apmserv5.2.6 php+mysql 
Navicat for MySQL

Recommended these two software, the installation is very simple, both are in Chinese, so it is easy for novices to operate.

Other things needed are a computer and about 10GB of hard drive space. A larger dictionary.

# Let’s start the first part, open the Navicat for MySQL connection, it’s very simple, I won’t take a screenshot here.

Recommendation: "mysql video tutorial"

Start creating the table, hash_cmd5, field, plaintext, cmd5_16, cmd5_32

are all set to not empty , the first is 255, the second is 16, and the third is 32 bits.

#Start importing data. It’s very simple. Just refer to the picture and figure it out yourself.

#Refer to your own database source and set the encoding

Refer to your own database content.

The fourth and fifth parts can be skipped directly.

Start the sixth step, you only need to set plaintext, other fields do not need to be set, but the prerequisite must be to select the plaintext found in the database, otherwise it will be meaningless.

#Go directly to step 8 to import data.

Let’s try it yourself. The next step is to teach you how to batch generate cmd16 32-bit.

High-speed single table import

If: your text file is in d:\aa.txt

Table name: t

Field name: c

is the following command

mysql> load data local infile 'd:/aa.txt' into table cmd5 lines terminated by ',' (cmd5_txt);

Create 32 16-bit md5 with one click

update `hash` set cmd5_16 = substr(md5(plaintext), 1, 16) where 1=1
 
update `hash` set cmd5_32 = md5(plaintext) where 1=1

As for some experts, this The method is unstable and so on. When tested on a single machine, there is no pressure on a single table of 600 million.

The above is the detailed content of About building billion-level cmd5 database with mysql and millisecond-level query (complete process). For more information, please follow other related articles on the PHP Chinese website!