PHP multi-process insert data
In the virtual machine centos7, single core, 1G memory
/**
* 模拟并发请求，10万次写入数据库
* 拆分为10个进程，每个进程处理一万条插入
*/
$total = 10000;
$num = 10;
$per = $total/$num;
$sql = '';
$child = '';
echo 'start '.microtime(true).PHP_EOL;
for($i = 1; $i<= $num; $i++)
{
$pid = pcntl_fork();
if($pid == -1) {
die('fork error');
}
if($pid > 0) {
//$id = pcntl_wait($status,WNOHANG);
$child[] = $pid;
} else if ($pid == 0) {
$link = mysqli_connect('localhost','root','root','yii2advanced');
$start = ($i-1)*$per + 1;
$end = $start + $per;
for($j = $start; $j< $end; $j++){
$time = microtime(true);
$sql = 'insert pcntl_test (rank,time) values ('.$j.','.$time.')';
mysqli_query($link,$sql);
}
mysqli_close($link);
$id = getmypid();
echo 'child '.$id.' finished '.microtime(true).PHP_EOL;
exit(0);
}
}
while(count($child)){
foreach($child as $k => $pid) {
$res = pcntl_waitpid($pid, $status, WNOHANG);
if ( -1 == $res || $res > 0) {
unset($child[$k]);
}
}
}
echo 'end '.microtime(true).PHP_EOL;When $total=10000, $num = 10; the execution result is as follows:
start 1491903371.5548 child 19860 finished 1491903417.2113 child 19857 finished 1491903417.6909 child 19864 finished 1491903417.7793 child 19855 finished 1491903417.8695 child 19859 finished 1491903417.9162 child 19861 finished 1491903418.0089 child 19856 finished 1491903418.0532 child 19863 finished 1491903418.0842 child 19862 finished 1491903418.1474 child 19858 finished 1491903418.4341 end 1491903418.4424 总时间为46.88759994506836秒
When $total=10000,$ When num = 100, the execution result is as follows:
start 1491904334.1735 child 20085 finished 1491904337.0712 child 20086 finished 1491904337.144 …… child 20262 finished 1491904341.5602 child 20264 finished 1491904341.5803 end 1491904341.5869 总时间为7.413399934768677
When $total=10000, $num = 1000, the execution result is as follows:
start 1491904562.0166 child 20282 finished 1491904562.1191 child 20277 finished 1491904562.1268 child 20279 finished 1491904562.1352 ... child 21586 finished 1491904576.6954 child 21582 finished 1491904576.7024 child 21584 finished 1491904576.7226 end 1491904576.7297 总时间为14.71310019493103，相比100个子进程，耗时更长了。进程切换太多，影响了了效率应该是原因之一。
When $total=100000, $num=100 , 100,000 records, 100 processes inserting
start 1491905670.2652 child 21647 finished 1491905725.4382 child 21651 finished 1491905725.4595 child 21642 finished 1491905725.5402 .... child 21810 finished 1491905729.7709 child 21812 finished 1491905729.8498 child 21811 finished 1491905729.9612 end 1491905729.9679 总时间为59.70270013809204
It takes 18 seconds for a single process to insert 10,000 records, which is less time-consuming than 10 processes inserting 10,000 records.
Inserting 100,000 records in a single process takes 187.40066790581, which is relatively slow. three minutes. . .
However, when I fork 1,000 processes to insert 100,000 records, an error may occur in about 36 seconds if successful. Mysqli_connection returns false. Is the number of connections limited?
Fork 10,000 child processes and insert 1 million data. At this time, there will be many connection errors. In the end, it took 360 seconds, and 945,300 records were inserted into the data table, with a success rate of 94.53%. So check the relevant configuration information of the database
mysql> show global status like '%connect%'; +-----------------------------------------------+---------------------+ | Variable_name | Value | +-----------------------------------------------+---------------------+ | Aborted_connects | 0 | | Connection_errors_accept | 0 | | Connection_errors_internal | 0 | | Connection_errors_max_connections | 628 | | Connection_errors_peer_address | 0 | | Connection_errors_select | 0 | | Connection_errors_tcpwrap | 0 | | Connections | 16519 | | Locked_connects | 0 | | Max_used_connections | 501 | | Max_used_connections_time | 2017-04-12 15:19:54 | | Performance_schema_session_connect_attrs_lost | 0 | | Ssl_client_connects | 0 | | Ssl_connect_renegotiates | 0 | | Ssl_finished_connects | 0 | | Threads_connected | 4 | +-----------------------------------------------+---------------------+ mysql> show global variables like '%connect%'; +-----------------------------------------------+--------------------+ | Variable_name | Value | +-----------------------------------------------+--------------------+ | character_set_connection | utf8mb4 | | collation_connection | utf8mb4_general_ci | | connect_timeout | 10 | | disconnect_on_expired_password | ON | | init_connect | | | max_connect_errors | 100 | | max_connections | 500 | | max_user_connections | 0 | | performance_schema_session_connect_attrs_size | 512 | +-----------------------------------------------+--------------------+ 修改 myqsql 配置文件，/etc/my.cnf 把max_connections 改为10000，然后重启mysql 实际MySQL服务器允许的最大连接数16384； 结果然并卵，虚拟机好像挂了了。
When the concurrency is large, the problem lies in connecting to mysql.
You can try to solve this problem through a connection pool.
In the virtual machine centos7, single core, 1G memory
/**
* 模拟并发请求，10万次写入数据库
* 拆分为10个进程，每个进程处理一万条插入
*/
$total = 10000;
$num = 10;
$per = $total/$num;
$sql = '';
$child = '';
echo 'start '.microtime(true).PHP_EOL;
for($i = 1; $i<= $num; $i++)
{
$pid = pcntl_fork();
if($pid == -1) {
die('fork error');
}
if($pid > 0) {
//$id = pcntl_wait($status,WNOHANG);
$child[] = $pid;
} else if ($pid == 0) {
$link = mysqli_connect('localhost','root','root','yii2advanced');
$start = ($i-1)*$per + 1;
$end = $start + $per;
for($j = $start; $j< $end; $j++){
$time = microtime(true);
$sql = 'insert pcntl_test (rank,time) values ('.$j.','.$time.')';
mysqli_query($link,$sql);
}
mysqli_close($link);
$id = getmypid();
echo 'child '.$id.' finished '.microtime(true).PHP_EOL;
exit(0);
}
}
while(count($child)){
foreach($child as $k => $pid) {
$res = pcntl_waitpid($pid, $status, WNOHANG);
if ( -1 == $res || $res > 0) {
unset($child[$k]);
}
}
}
echo 'end '.microtime(true).PHP_EOL;When $total=10000, $num = 10; the execution result is as follows:
start 1491903371.5548 child 19860 finished 1491903417.2113 child 19857 finished 1491903417.6909 child 19864 finished 1491903417.7793 child 19855 finished 1491903417.8695 child 19859 finished 1491903417.9162 child 19861 finished 1491903418.0089 child 19856 finished 1491903418.0532 child 19863 finished 1491903418.0842 child 19862 finished 1491903418.1474 child 19858 finished 1491903418.4341 end 1491903418.4424 总时间为46.88759994506836秒
When $total= 10000, $num = 100, the execution result is as follows:
start 1491904334.1735 child 20085 finished 1491904337.0712 child 20086 finished 1491904337.144 …… child 20262 finished 1491904341.5602 child 20264 finished 1491904341.5803 end 1491904341.5869 总时间为7.413399934768677
When $total=10000,$num = 1000, the execution result is as follows:
start 1491904562.0166 child 20282 finished 1491904562.1191 child 20277 finished 1491904562.1268 child 20279 finished 1491904562.1352 ... child 21586 finished 1491904576.6954 child 21582 finished 1491904576.7024 child 21584 finished 1491904576.7226 end 1491904576.7297 总时间为14.71310019493103，相比100个子进程，耗时更长了。进程切换太多，影响了了效率应该是原因之一。
When $total=100000,$num =100, 100,000 records, 100 processes inserting
start 1491905670.2652 child 21647 finished 1491905725.4382 child 21651 finished 1491905725.4595 child 21642 finished 1491905725.5402 .... child 21810 finished 1491905729.7709 child 21812 finished 1491905729.8498 child 21811 finished 1491905729.9612 end 1491905729.9679 总时间为59.70270013809204
It takes 18 seconds for a single process to insert 10,000 records, which is less time-consuming than 10 processes inserting 10,000 records.
Inserting 100,000 records in a single process takes 187.40066790581, which is relatively slow. three minutes. . .
However, when I fork 1,000 processes to insert 100,000 records, an error may occur in about 36 seconds if successful. Mysqli_connection returns false. Is the number of connections limited?
Fork 10,000 sub-processes and insert 1 million data. At this time, there will be many connection errors. In the end, it took 360 seconds, and 945,300 records were inserted into the data table, with a success rate of 94.53%. So check the relevant configuration information of the database
mysql> show global status like '%connect%'; +-----------------------------------------------+---------------------+ | Variable_name | Value | +-----------------------------------------------+---------------------+ | Aborted_connects | 0 | | Connection_errors_accept | 0 | | Connection_errors_internal | 0 | | Connection_errors_max_connections | 628 | | Connection_errors_peer_address | 0 | | Connection_errors_select | 0 | | Connection_errors_tcpwrap | 0 | | Connections | 16519 | | Locked_connects | 0 | | Max_used_connections | 501 | | Max_used_connections_time | 2017-04-12 15:19:54 | | Performance_schema_session_connect_attrs_lost | 0 | | Ssl_client_connects | 0 | | Ssl_connect_renegotiates | 0 | | Ssl_finished_connects | 0 | | Threads_connected | 4 | +-----------------------------------------------+---------------------+ mysql> show global variables like '%connect%'; +-----------------------------------------------+--------------------+ | Variable_name | Value | +-----------------------------------------------+--------------------+ | character_set_connection | utf8mb4 | | collation_connection | utf8mb4_general_ci | | connect_timeout | 10 | | disconnect_on_expired_password | ON | | init_connect | | | max_connect_errors | 100 | | max_connections | 500 | | max_user_connections | 0 | | performance_schema_session_connect_attrs_size | 512 | +-----------------------------------------------+--------------------+ 修改 myqsql 配置文件，/etc/my.cnf 把max_connections 改为10000，然后重启mysql 实际MySQL服务器允许的最大连接数16384； 结果然并卵，虚拟机好像挂了了。
When the concurrency is large, the problem lies in connecting to mysql.
You can try to solve this problem through a connection pool.
The above is the detailed content of PHP multi-process insert data. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1664
14
1423
52
1317
25
1268
29
1246
24
Explain JSON Web Tokens (JWT) and their use case in PHP APIs.
Apr 05, 2025 am 12:04 AM
JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,
PHP Program to Count Vowels in a String
Feb 07, 2025 pm 12:12 PM
A string is a sequence of characters, including letters, numbers, and symbols. This tutorial will learn how to calculate the number of vowels in a given string in PHP using different methods. The vowels in English are a, e, i, o, u, and they can be uppercase or lowercase. What is a vowel? Vowels are alphabetic characters that represent a specific pronunciation. There are five vowels in English, including uppercase and lowercase: a, e, i, o, u Example 1 Input: String = "Tutorialspoint" Output: 6 explain The vowels in the string "Tutorialspoint" are u, o, i, a, o, i. There are 6 yuan in total
Explain late static binding in PHP (static::).
Apr 03, 2025 am 12:04 AM
Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.
What are PHP magic methods (__construct, __destruct, __call, __get, __set, etc.) and provide use cases?
Apr 03, 2025 am 12:03 AM
What are the magic methods of PHP? PHP's magic methods include: 1.\_\_construct, used to initialize objects; 2.\_\_destruct, used to clean up resources; 3.\_\_call, handle non-existent method calls; 4.\_\_get, implement dynamic attribute access; 5.\_\_set, implement dynamic attribute settings. These methods are automatically called in certain situations, improving code flexibility and efficiency.
PHP and Python: Comparing Two Popular Programming Languages
Apr 14, 2025 am 12:13 AM
PHP and Python each have their own advantages, and choose according to project requirements. 1.PHP is suitable for web development, especially for rapid development and maintenance of websites. 2. Python is suitable for data science, machine learning and artificial intelligence, with concise syntax and suitable for beginners.
PHP in Action: Real-World Examples and Applications
Apr 14, 2025 am 12:19 AM
PHP is widely used in e-commerce, content management systems and API development. 1) E-commerce: used for shopping cart function and payment processing. 2) Content management system: used for dynamic content generation and user management. 3) API development: used for RESTful API development and API security. Through performance optimization and best practices, the efficiency and maintainability of PHP applications are improved.
PHP: A Key Language for Web Development
Apr 13, 2025 am 12:08 AM
PHP is a scripting language widely used on the server side, especially suitable for web development. 1.PHP can embed HTML, process HTTP requests and responses, and supports a variety of databases. 2.PHP is used to generate dynamic web content, process form data, access databases, etc., with strong community support and open source resources. 3. PHP is an interpreted language, and the execution process includes lexical analysis, grammatical analysis, compilation and execution. 4.PHP can be combined with MySQL for advanced applications such as user registration systems. 5. When debugging PHP, you can use functions such as error_reporting() and var_dump(). 6. Optimize PHP code to use caching mechanisms, optimize database queries and use built-in functions. 7
The Enduring Relevance of PHP: Is It Still Alive?
Apr 14, 2025 am 12:12 AM
PHP is still dynamic and still occupies an important position in the field of modern programming. 1) PHP's simplicity and powerful community support make it widely used in web development; 2) Its flexibility and stability make it outstanding in handling web forms, database operations and file processing; 3) PHP is constantly evolving and optimizing, suitable for beginners and experienced developers.
