Table of Contents
项目地址
功能
设计
InitCollection
CopyData
OplogSync
问题
目前进度 :
Home Database Mysql Tutorial mongodb迁移工具

mongodb迁移工具

Jun 07, 2016 pm 04:41 PM
github mongodb address tool migrate project

项目地址 [github]地址戳这里.(https://github.com/xbsura/mongo-migrate) 使用go语言编写,与m-tools保持一致,除了go语言自带的库,还需要mgo库支持. 功能 mongodb实例之间的数据迁移,带有根据源与目的的节点性质,进行自动分片,预split与movechunk,创建索引,

项目地址

[github]地址戳这里.(https://github.com/xbsura/mongo-migrate)

使用go语言编写,与m-tools保持一致,除了go语言自带的库,还需要mgo库支持.

功能

mongodb实例之间的数据迁移,带有根据源与目的的节点性质,进行自动分片,预split与movechunk,创建索引,拷贝数据与同步oplog几个过程.

设计

迁移工具分为三个阶段,初始化阶段,数据拷贝与oplog同步,分别使用InitCollection类,CopyData类与OplogSync类实现.

InitCollection

过程:

  1. 建立与源,目标的连接,若失败,raise.
  2. 初始化两个标识位,s(shard),o(oplog),为0, 0.
  3. 检测源类型,若为mongod,检测是否为复制集,若无法检测,raise;若为复制集,令o=1;若为mongos,查看要迁移的ns是否分片,如果分片,检测数据分片类型,若都为复制集,令o=1.
  4. 检测目标类型,若为mongod,令s=0.
  5. 若s=1,在目标进行分片,按照源的chunk进行split并随机movechunk.

CopyData

过程:
1. 建立源与目的的连接.
2. 获取要迁移的ns.
4. 如果o=1,记录源每个数据分片oplog最新值.
5. 如果源ns分片,按照chunk边界从源并发读,并向目标并发写;如果源未分片,使用一个query查询数据并向目标写.

OplogSync

过程:
1. 如果o=0,退出.
2. 建立源各个数据节点,目标节点(单节点)的连接.
3. 在每个源分片查找oplog ts大于等于CopyData记录时间点的记录,并逐条应用到目标节点.

问题

  1. 日志貌似打印有一点点问题.
  2. 性能没测试.
  3. 代码有点烂.

目前进度 :

完成可运行的版本,没有完整测试.

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1659
14
PHP Tutorial
1258
29
C# Tutorial
1232
24
How to use the chrono library in C? How to use the chrono library in C? Apr 28, 2025 pm 10:18 PM

Using the chrono library in C can allow you to control time and time intervals more accurately. Let's explore the charm of this library. C's chrono library is part of the standard library, which provides a modern way to deal with time and time intervals. For programmers who have suffered from time.h and ctime, chrono is undoubtedly a boon. It not only improves the readability and maintainability of the code, but also provides higher accuracy and flexibility. Let's start with the basics. The chrono library mainly includes the following key components: std::chrono::system_clock: represents the system clock, used to obtain the current time. std::chron

How to understand DMA operations in C? How to understand DMA operations in C? Apr 28, 2025 pm 10:09 PM

DMA in C refers to DirectMemoryAccess, a direct memory access technology, allowing hardware devices to directly transmit data to memory without CPU intervention. 1) DMA operation is highly dependent on hardware devices and drivers, and the implementation method varies from system to system. 2) Direct access to memory may bring security risks, and the correctness and security of the code must be ensured. 3) DMA can improve performance, but improper use may lead to degradation of system performance. Through practice and learning, we can master the skills of using DMA and maximize its effectiveness in scenarios such as high-speed data transmission and real-time signal processing.

How to measure thread performance in C? How to measure thread performance in C? Apr 28, 2025 pm 10:21 PM

Measuring thread performance in C can use the timing tools, performance analysis tools, and custom timers in the standard library. 1. Use the library to measure execution time. 2. Use gprof for performance analysis. The steps include adding the -pg option during compilation, running the program to generate a gmon.out file, and generating a performance report. 3. Use Valgrind's Callgrind module to perform more detailed analysis. The steps include running the program to generate the callgrind.out file and viewing the results using kcachegrind. 4. Custom timers can flexibly measure the execution time of a specific code segment. These methods help to fully understand thread performance and optimize code.

How to use string streams in C? How to use string streams in C? Apr 28, 2025 pm 09:12 PM

The main steps and precautions for using string streams in C are as follows: 1. Create an output string stream and convert data, such as converting integers into strings. 2. Apply to serialization of complex data structures, such as converting vector into strings. 3. Pay attention to performance issues and avoid frequent use of string streams when processing large amounts of data. You can consider using the append method of std::string. 4. Pay attention to memory management and avoid frequent creation and destruction of string stream objects. You can reuse or use std::stringstream.

What is real-time operating system programming in C? What is real-time operating system programming in C? Apr 28, 2025 pm 10:15 PM

C performs well in real-time operating system (RTOS) programming, providing efficient execution efficiency and precise time management. 1) C Meet the needs of RTOS through direct operation of hardware resources and efficient memory management. 2) Using object-oriented features, C can design a flexible task scheduling system. 3) C supports efficient interrupt processing, but dynamic memory allocation and exception processing must be avoided to ensure real-time. 4) Template programming and inline functions help in performance optimization. 5) In practical applications, C can be used to implement an efficient logging system.

What kind of software is a digital currency app? Top 10 Apps for Digital Currencies in the World What kind of software is a digital currency app? Top 10 Apps for Digital Currencies in the World Apr 30, 2025 pm 07:06 PM

With the popularization and development of digital currency, more and more people are beginning to pay attention to and use digital currency apps. These applications provide users with a convenient way to manage and trade digital assets. So, what kind of software is a digital currency app? Let us have an in-depth understanding and take stock of the top ten digital currency apps in the world.

How to optimize code How to optimize code Apr 28, 2025 pm 10:27 PM

C code optimization can be achieved through the following strategies: 1. Manually manage memory for optimization use; 2. Write code that complies with compiler optimization rules; 3. Select appropriate algorithms and data structures; 4. Use inline functions to reduce call overhead; 5. Apply template metaprogramming to optimize at compile time; 6. Avoid unnecessary copying, use moving semantics and reference parameters; 7. Use const correctly to help compiler optimization; 8. Select appropriate data structures, such as std::vector.

How to uninstall MySQL and clean residual files How to uninstall MySQL and clean residual files Apr 29, 2025 pm 04:03 PM

To safely and thoroughly uninstall MySQL and clean all residual files, follow the following steps: 1. Stop MySQL service; 2. Uninstall MySQL packages; 3. Clean configuration files and data directories; 4. Verify that the uninstallation is thorough.

See all articles