Introduction to Redis batch writing

The following column Redis Tutorial will introduce you to Redis batch writing. I hope it will be helpful to friends in need!

Recently tested the performance of redis and needed to insert more than 10 million pieces of data in batches.

simply conducted a study and there are probably the following methods:

1: Java program call, simple for loop, direct insertion through Jedis method,

As for speed, you don’t need to look at it, don’t try it, if it doesn’t work at all, it won’t be implemented.

Two: Through shell script, it is relatively simple, but it is also because it needs to be connected to redis through the port.

The speed is also very slow, so give up.

Three: Provide pipelines through redis. I feel that this method is the most reliable. Here is the implementation:

Step 1: First simply write a shell script:

for((i=1;i<=1000000;i++))        doecho "set k$i v$i" >> /tmp/_t.txt        done

The purpose is to generate a script file for batch insertion.

Step 2: The file generated on Linux or Windows cannot be run directly as a redis statement.

To put it simply, because of Linux, The line breaks of windows and dos are different,

so you need to do a simple conversion:

There are many conversion methods:

1: In general Linux distributions All come with this small tool, which can only convert DOS to UNIX files. The command is as follows:

# unix2dos dosfile.txt

The above command will remove the ^M symbol at the end of the line. (If nuix2dos is not installed on the machine, just use method 2)

2: Use vim, vim is an improved version of vi, usage method:

#vim file.txt
:set fileformat=dos
::wq

One line command , save and exit,

Step 3: Just run the script, through the pipe provided by redis--pipe form,

cat redisTest.txt | redis- cli -h reids.aliyuncs.com -p 6379 -a xxxx --pipe

Three steps are enough to complete the batch insertion of redis. The speed is, in the case of 20 million records, The speed of generating scripts is a little slow, and the insertion speed is still a few minutes.

The speed can be said to be very fast~

The above is the detailed content of Introduction to Redis batch writing. For more information, please follow other related articles on the PHP Chinese website!