PHP sort 100 million numbers

Disassemble into thousands of parts, sort and merge them again.


First create a txt with 100 million QQ numbers.

PHP code?

//Create txt of 100 million QQ numbers (about 85~100 seconds)

set_time_limit (0);

$fn = 'qq.txt';

$fp = fopen($fn, 'w');

$st = microtime(true);

$l = range(0,10000 );

shuffle($l);

foreach ($l as $k=>$v)

{

$arr = range($v*10000+10000,10000*($v+1) +9999);

shuffle($arr);

fputs($fp,implode("n", $arr)."n");

unset($arr);

}

echo microtime(true)-$st;

?>

Wait a minute or two for 100 million random QQs to be created.

QQ number range is >10000. The file size is approximately 840MB.



The following will be classified into thousands of documents.

The length of the QQ number is used as the folder, and the first 3 digits of the QQ number are used as the file name.

PHP code?

//Length number classification (about 360~400 seconds)

set_time_limit(0);

$st = microtime (true);

if(!is_dir('qq_no')) mkdir('qq_no');

$file = fopen('qq.txt', 'r');

$ i=0;

$end_s = '';

while(!feof($file))

{

$g = 1042*1024;

fseek($file,$g*$i);

$s = fread($file, $g);

$end = strrpos($s, "n");

$arr_s = $end_s.substr($s, 0, $end);

$end_s = substr($s, $end);

$arr = explode("n", $arr_s);

foreach ($arr as $k=>$v)

{

if ($ v! = '') {

$ tag = "$ v [0] $ v [1] $ v [2]";

$ text_arr [strlen ($ v)] [$ tag] [] = $v;

}

}

foreach ($text_arr as $k=>$v)

{

 $n_dir = 'qq_no /'.$k;

    if (! is_dir($n_dir)) mkdir($n_dir); txt ', 'a+');

+ +$i;

}

echo microtime(true)-$st;

?>

Finally, each file is sorted and merged.

PHP code?

// Sorting completed (800~920 seconds)

set_time _limit(0);

$st = microtime(true);

$qq_done = fopen('qq_done.txt', 'a+');

$root = 'qq_no';

$dir_array = scandir($root);

foreach ($dir_array as $key=>$val)

{

if ($val != '.' && $val != '..')

$dirs[$val] = scandir($ root.'/'.$val);

}

foreach ($dirs as $key=>$val)

{

foreach ($val as $v)

    {

        if ($v != '.' && $v != '..')

        {

            $file = $root. '/' . $key . '/'. $v;

            $c = file_get_contents($file);

            $arr = explode("n", $c);

            sort($arr);

            fputs($qq_done, implode("n",$arr));

            unlink($file);

        }

    }

    rmdir($root. '/' . $key);

}

rmdir($root);

 

echo  microtime(true)-$st;

 

?>