Data table migration data consistency verification
It is still very useful to do some necessary verification when migrating the database. For example, whether the number of data items before and after migration is consistent, and whether the data is consistent. What should I do at this time? It is easy to verify the number of items, but what if I verify whether the data is consistent? For important data, of course, every piece of data must be error-free. Random sampling verification is definitely not possible. If it is missed, it will be troublesome, and the two tables are no longer on the same server. What to do at this time? There is a way:
- Select a few important fields from the table, such as A, B, and C, and use these fields as a comparison ruler.
- Export these three fields of each piece of data from the original table to a file f1.
- Transfer these three fields of each piece of data from the destination table to file f2.
- Compare each piece of data in files f1 and f2 to see if they are the same.
- Draw a conclusion
The above method was thought up at the same time, and it is not bad, but I think there is room for improvement:
- First of all, not all fields may still appear different in non-main fields.
- The overall efficiency is relatively low
My thoughts are this:
- Splice every n pieces of data in the table (connect directly, the value of n depends on the size of each piece of data).
- Calculate the md5 values of these n pieces of data and add them to file f1 until all data values are obtained.
- The same goes for the destination table, which is recorded in file f2.
- Compare the md5 values of files f1 and f2. If they are consistent, ok, success.
- If they are inconsistent, compare each md5 value from top to bottom and find the mth inconsistent value.
- Concluded that the inconsistent data is between m*(n-1)+1 ~ m*n, and you can choose positioning again.
The advantage of the second method is that the output file will be reduced within a certain range, making it easy to compare. However, it also has disadvantages. It cannot directly locate the location of different data through key fields like the first method.
The following is the specific code implementation of the second method’s effect sum:
<?php
/**
* 使用方法：
* php -f mysql_diff.php yes dir 10
* 是否计算条数 是否计算输出d5并保存到文件 合并数据的级别
*
*/
if(php_sapi_name() != 'cli')
{
die("请在CLI模式下运行");
}
array_shift($argv);
if(empty($argv))
{
die("at letase contain one info");
}
$is_count = array_shift($argv);
$is_md5 = empty($argv) ? false : array_shift($argv);
$conbine_num = empty($argv) ? 1 : intval(array_shift($argv));
if($is_md5 && !is_dir($is_md5) && !mkdir($is_md5, 777, true))
{
die("error info : md5 info must be input to a file");
}
$dbinfos = array(
'host' => 'localhost',
'port' => '3306',
'user' => 'root',
'pswd' => '123456',
'charset' => 'utf8',
'tables' => array(
'lagou.pos',
'lagou.pos_innodb',
),
);
//验证格式
if(!$link = mysql_connect($dbinfos['host'].":".$dbinfos['port'],$dbinfos['user'], $dbinfos['pswd']))
{
die("connect to [{$host}@{$port}] failed!!");
}
if(!mysql_query("set names {$dbinfos['charset']}"))
{
die("set charset error : ".mysql_error());
}
foreach ($dbinfos['tables'] as $table)
{
if($is_count)
{
$sql = "select count(*) as nums from {$table}";
$ret = mysql_query($sql);
if(!$ret)
{
die("error : ".mysql_error());
}
$ret = mysql_fetch_array($ret, MYSQL_ASSOC);
echo "{$table} : {$ret['nums']}\n";
}
if($is_md5)
{
$path = $is_md5.DIRECTORY_SEPARATOR.$table;
$sql = "select * from {$table}";
$ret = mysql_query($sql);
$flag = 0;
$fields = '';
while ($_ret = mysql_fetch_array($ret, MYSQL_NUM)) {
$flag++;
while($_ret)
{
$fields .= array_pop($_ret);
}
if($flag % $conbine_num == 0)
{
file_put_contents($path, md5($fields)."\n", FILE_APPEND);
$fields = '';
}
}
if($flag % $conbine_num != 0 && $flag > 0)
{
file_put_contents($path, md5($fields)."\n", FILE_APPEND);
}
echo "save to file info : ".realpath($path)."\n";
}
}
The above introduces the data consistency verification of data table migration, including aspects of the content. I hope it will be helpful to friends who are interested in PHP tutorials.
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
1664
14
1423
52
1317
25
1268
29
1246
24
Laravel Introduction Example
Apr 18, 2025 pm 12:45 PM
Laravel is a PHP framework for easy building of web applications. It provides a range of powerful features including: Installation: Install the Laravel CLI globally with Composer and create applications in the project directory. Routing: Define the relationship between the URL and the handler in routes/web.php. View: Create a view in resources/views to render the application's interface. Database Integration: Provides out-of-the-box integration with databases such as MySQL and uses migration to create and modify tables. Model and Controller: The model represents the database entity and the controller processes HTTP requests.
MySQL and phpMyAdmin: Core Features and Functions
Apr 22, 2025 am 12:12 AM
MySQL and phpMyAdmin are powerful database management tools. 1) MySQL is used to create databases and tables, and to execute DML and SQL queries. 2) phpMyAdmin provides an intuitive interface for database management, table structure management, data operations and user permission management.
MySQL vs. Other Programming Languages: A Comparison
Apr 19, 2025 am 12:22 AM
Compared with other programming languages, MySQL is mainly used to store and manage data, while other languages such as Python, Java, and C are used for logical processing and application development. MySQL is known for its high performance, scalability and cross-platform support, suitable for data management needs, while other languages have advantages in their respective fields such as data analytics, enterprise applications, and system programming.
Solve database connection problem: a practical case of using minii/db library
Apr 18, 2025 am 07:09 AM
I encountered a tricky problem when developing a small application: the need to quickly integrate a lightweight database operation library. After trying multiple libraries, I found that they either have too much functionality or are not very compatible. Eventually, I found minii/db, a simplified version based on Yii2 that solved my problem perfectly.
Laravel framework installation method
Apr 18, 2025 pm 12:54 PM
Article summary: This article provides detailed step-by-step instructions to guide readers on how to easily install the Laravel framework. Laravel is a powerful PHP framework that speeds up the development process of web applications. This tutorial covers the installation process from system requirements to configuring databases and setting up routing. By following these steps, readers can quickly and efficiently lay a solid foundation for their Laravel project.
MySQL for Beginners: Getting Started with Database Management
Apr 18, 2025 am 12:10 AM
The basic operations of MySQL include creating databases, tables, and using SQL to perform CRUD operations on data. 1. Create a database: CREATEDATABASEmy_first_db; 2. Create a table: CREATETABLEbooks(idINTAUTO_INCREMENTPRIMARYKEY, titleVARCHAR(100)NOTNULL, authorVARCHAR(100)NOTNULL, published_yearINT); 3. Insert data: INSERTINTObooks(title, author, published_year)VA
Solve MySQL mode problem: The experience of using the TheliaMySQLModesChecker module
Apr 18, 2025 am 08:42 AM
When developing an e-commerce website using Thelia, I encountered a tricky problem: MySQL mode is not set properly, causing some features to not function properly. After some exploration, I found a module called TheliaMySQLModesChecker, which is able to automatically fix the MySQL pattern required by Thelia, completely solving my troubles.
MySQL: Structured Data and Relational Databases
Apr 18, 2025 am 12:22 AM
MySQL efficiently manages structured data through table structure and SQL query, and implements inter-table relationships through foreign keys. 1. Define the data format and type when creating a table. 2. Use foreign keys to establish relationships between tables. 3. Improve performance through indexing and query optimization. 4. Regularly backup and monitor databases to ensure data security and performance optimization.
