Home Database Mysql Tutorial mysql中百万级数据插入速度测试

mysql中百万级数据插入速度测试

Jun 07, 2016 pm 05:51 PM
mysql nbsp quot thumb

100W的数据对于很多朋友来说算小菜了,但今天我就想到创建一个大量数据的测试环境,于是找了一下怎么插入100W条数据,我用的是20个字段。对比一下,首先是用 mysql 的存储过程弄的

100W的数据对于很多朋友来说算小菜了,但今天我就想到创建一个大量数据的测试环境,于是找了一下怎么插入100W条数据,我用的是20个字段。对比一下,首先是用 的存储过程弄的:

 代码如下 复制代码

mysql>delimiter $
mysql>SET AUTOCOMMIT = 0$$

mysql> create  procedure test()
begin
declare i decimal (10)  default 0 ;
dd:loop       
INSERT INTO `million` (`categ_id`, `categ_fid`, `SortPath`, `address`, `p_identifier`, `pro_specification`, `name`, `add_date`, `picture_url`, `thumb_url`, `is_display_front`, `create_html_time`, `hit`, `buy_sum`, `athor`, `templete _style`, `is_hot`, `is_new`, `is_best`) VALUES
(268, 2, '0,262,268,', 0, '2342', '423423', '123123', '2012-01-09 09:55:43', 'upload/product/20111205153432_53211.jpg', 'upload/product/thumb_20111205153432_53211.jpg', 1, 0, 0, 0, 'admin', '0', 0, 0, 0);
   commit;
   set i = i+1;
   if i= 1000000 then leave dd;
    end if;
   end loop dd ;
end;$

mysql>delimiter ;

mysql> call test;
 

结果我们看了用了58分钟,这也太费时差了吧

mysql>  call test;
Query OK, 0 rows affected (58 min 30.83 sec)
非常耗时。

于是我又找了一个方法

先用PHP代码生成数据,再导入:

 代码如下 复制代码


$t=mktime();
set_time_limit(1000);
$myFile="e:/insert.sql";
$fhandler=($myFile,'wb');
if($fhandler){

    $sql="268t2t'0,262,268,'t0t '2342't'423423't'123123't'23423423't'2012-01-09 09:55:43't'upload/product/20111205153432_53211.jpg't'upload/product/thumb_20111205153432_53211.jpg'tNULLtNULLt38t'件't''t123t123t0";
    $i=0;
    while($i     {
        $i++;
        fwrite($fhandler,$sql."rn");
    }
    echo"写入成功,耗时:",mktime()-$t;
}

然后再导入

 代码如下 复制代码
LOAD DATA local INFILE 'e:/insert.sql' INTO TABLE  tenmillion(`categ_id`, `categ_fid`, `SortPath`, `address`, `p_identifier`, `pro_specification`, `name`, `description`, `add_date`, `picture_url`, `thumb_url`, `shop_url`, `shop_thumb_url`, `brand_id`, `unit`, `square_meters_unit`, `market_price`, `true_price`, `square_meters_price`);


结果不到1分钟,100万的数据就快速的导入了,

注意字段不再以逗号分割,以t分割,条记录以rn分割。结果我插入10次数据,100W平均只要1分钟搞定。

总结,在大数据量处理时我们最好利用第三方插件一实现数据备份或直接在服务器上进行备份,用mysql自带的工具有时确实不理想。

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

How to open phpmyadmin How to open phpmyadmin Apr 10, 2025 pm 10:51 PM

You can open phpMyAdmin through the following steps: 1. Log in to the website control panel; 2. Find and click the phpMyAdmin icon; 3. Enter MySQL credentials; 4. Click "Login".

MySQL: An Introduction to the World's Most Popular Database MySQL: An Introduction to the World's Most Popular Database Apr 12, 2025 am 12:18 AM

MySQL is an open source relational database management system, mainly used to store and retrieve data quickly and reliably. Its working principle includes client requests, query resolution, execution of queries and return results. Examples of usage include creating tables, inserting and querying data, and advanced features such as JOIN operations. Common errors involve SQL syntax, data types, and permissions, and optimization suggestions include the use of indexes, optimized queries, and partitioning of tables.

How to use single threaded redis How to use single threaded redis Apr 10, 2025 pm 07:12 PM

Redis uses a single threaded architecture to provide high performance, simplicity, and consistency. It utilizes I/O multiplexing, event loops, non-blocking I/O, and shared memory to improve concurrency, but with limitations of concurrency limitations, single point of failure, and unsuitable for write-intensive workloads.

MySQL's Place: Databases and Programming MySQL's Place: Databases and Programming Apr 13, 2025 am 12:18 AM

MySQL's position in databases and programming is very important. It is an open source relational database management system that is widely used in various application scenarios. 1) MySQL provides efficient data storage, organization and retrieval functions, supporting Web, mobile and enterprise-level systems. 2) It uses a client-server architecture, supports multiple storage engines and index optimization. 3) Basic usages include creating tables and inserting data, and advanced usages involve multi-table JOINs and complex queries. 4) Frequently asked questions such as SQL syntax errors and performance issues can be debugged through the EXPLAIN command and slow query log. 5) Performance optimization methods include rational use of indexes, optimized query and use of caches. Best practices include using transactions and PreparedStatemen

Why Use MySQL? Benefits and Advantages Why Use MySQL? Benefits and Advantages Apr 12, 2025 am 12:17 AM

MySQL is chosen for its performance, reliability, ease of use, and community support. 1.MySQL provides efficient data storage and retrieval functions, supporting multiple data types and advanced query operations. 2. Adopt client-server architecture and multiple storage engines to support transaction and query optimization. 3. Easy to use, supports a variety of operating systems and programming languages. 4. Have strong community support and provide rich resources and solutions.

Monitor Redis Droplet with Redis Exporter Service Monitor Redis Droplet with Redis Exporter Service Apr 10, 2025 pm 01:36 PM

Effective monitoring of Redis databases is critical to maintaining optimal performance, identifying potential bottlenecks, and ensuring overall system reliability. Redis Exporter Service is a powerful utility designed to monitor Redis databases using Prometheus. This tutorial will guide you through the complete setup and configuration of Redis Exporter Service, ensuring you seamlessly build monitoring solutions. By studying this tutorial, you will achieve fully operational monitoring settings

How to view sql database error How to view sql database error Apr 10, 2025 pm 12:09 PM

The methods for viewing SQL database errors are: 1. View error messages directly; 2. Use SHOW ERRORS and SHOW WARNINGS commands; 3. Access the error log; 4. Use error codes to find the cause of the error; 5. Check the database connection and query syntax; 6. Use debugging tools.

How to connect to the database of apache How to connect to the database of apache Apr 13, 2025 pm 01:03 PM

Apache connects to a database requires the following steps: Install the database driver. Configure the web.xml file to create a connection pool. Create a JDBC data source and specify the connection settings. Use the JDBC API to access the database from Java code, including getting connections, creating statements, binding parameters, executing queries or updates, and processing results.

See all articles