Implementation code for excluding duplicate data from millions of data in PHP array

If you get a uid list with more than one million rows, the format is as follows:

Copy the code The code is as follows:

 
10001000 
10001001 
10001002 
...... 
10001000 
...... 
10001111　To sort out the duplication, let’s first take a look at the definition of a PHP array: an array in PHP is actually an ordered map. A map is a type that associates values ​​to keys. This type is optimized in many ways, so it can be treated as a real array, or a list (vector), a hash table (an implementation of a map), a dictionary, a set, a stack, a queue, and many more possibilities. The value of an array element can also be another array. Tree structures and multidimensional arrays are also allowed. 

In PHP arrays, keys are also called indexes and are unique. We can use this feature to perform deduplication. The sample code is as follows:

Copy the code

The code is as follows: //Define an array to store the results after deduplication

$result = array (); 
//Read uid list file
$fp = fopen('test.txt', 'r'); 

while(!feof($fp)) 
 { 
$uid = fgets($fp); 
$uid = trim($uid); 
$uid = trim($uid, "r"); 
$uid = trim($ uid, "n"); 

if($uid == '') 
{ 
continue; 
} 
//Us uid as the key to see if the value exists 
if(empty($result[$uid])) 
{ 
$result[$uid] = 1; 
} 
} 

fclose($fp) ; 

//Save the result to file 
$content = ''; 
foreach($result as $k => $v) 
{ 
$content .= $k."n"; 
} 
$fp = fopen('result.txt', 'w'); 
fwrite($fp, $content); 
fclose($fp) ; 
?>　 

 
With more than 20 lines of code, more than one million data can be deduplicated. The efficiency is also good and very practical. Mobile phone numbers and emails can also be deduplicated in this way. 

Also, this method can also be used to deduplicate two files. If you have two uid list files, the format is the same as the uid list above. The sample program is as follows:

Copy the code

The code is as follows: //Define an array to store the results after deduplication

$result = array(); 
//Read the first uid list file and put $result_1 
$fp = fopen('test_1.txt', 'r'); 
while( !feof($fp)) 
{ 
$uid = fgets($fp); 
$uid = trim($uid); 
$uid = trim($uid, "r") ; 
$uid = trim($uid, "n"); 
if($uid == '') 
{ 
continue; 
} 
//Uid is The key is written to $result. If there is a duplicate, it will be overwritten. 
$result[$uid] = 1; 
} 
fclose($fp); 
//Read the second uid list file , and perform deduplication operation 
$fp = fopen('test_2.txt', 'r'); 
while(!feof($fp)) 
{ 
$uid = fgets($ fp); 
$uid = trim($uid); 
$uid = trim($uid, "r"); 
$uid = trim($uid, "n"); 
 if($uid == '') 
{ 
continue; 
} 
//Us uid as the key to see if the value exists 
if(empty($result[$uid] )) 
{ 
$result[$uid] = 1; 
} 
} 
fclose($fp); 
//What is saved in $result is after deduplication The result can be output to a file, and the code is omitted 
?> 

 
If you think about it carefully, it is not difficult to find that using this feature of arrays can solve more problems in our work.

http://www.bkjia.com/PHPjc/321981.html

Related labels：

php uid code list exist accomplish right exclude data array quantity million of conduct repeat

source：php.cn

Previous article：PHP XML error parsing SOAP payload on line 1_PHP教程 Next article：PHP cache implementation code and detailed comments_PHP tutorial

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Implementation code for excluding duplicate data from millions of data in PHP array_PHP tutorial