Database cloudification: realizing software-defined storage

Introduction

Software-defined storage is the development trend of the industry, which can ensure that the storage access of the system is managed more flexibly at a precise level. This article shares the database cloudization practice of software-defined storage, which was shared and compiled from the lecture hall last Thursday. .

What is Software Defined Storage (SDS)

Software-defined storage (SDS) is a data storage method in which all storage-related control work is placed in external software relative to the physical storage hardware. This software is not available as firmware on a storage device, but on a server or as part of the operating system (OS) or hypervisor.

Software-defined storage is an industry development trend, which can ensure that system storage access is more flexibly managed at a precise level and abstract software from hardware storage, which means it can become a system that is not restricted by physical systems. Shared pool for more efficient use of resources. Storage can also be deployed and provisioned through software and management, and its management procedures can be further simplified through automated policy-based management.

What are the characteristics of software-defined storage

Scalable. The software-defined storage system can be expanded to a cluster size of hundreds or even thousands of units. Moreover, as the cluster size increases, the overall performance of the system increases linearly
low cost. The automatic fault tolerance and automatic load balancing mechanism of the software-defined storage system enable it to be built on ordinary X86 servers. In addition, the linear expansion capability also makes it very convenient to add and reduce machines, and can realize automatic operation and maintenance.
high performance. Whether for an entire cluster or a single server, distributed storage systems offer high performance.
Traditional IT architecture model:

Historically, traditional architecture databases have been running on minicomputers and centralized SAN storage. With the rapid development of business, the transaction volume and data volume of business systems are increasing, and the requirements for the processing and storage capabilities of business systems are getting higher and higher. The traditional architecture has poor scalability, limited computing power, and a serious lack of IO capabilities. To achieve the corresponding processing capabilities, it is necessary to replace the existing minicomputers and storage with higher-configuration minicomputers and higher-end storage, which is technically complex. Expensive. At the same time, as equipment ages, equipment performance is increasingly insufficient to support business needs, and business system architecture transformation and upgrades are imperative.

Some disadvantages of traditional architecture:

Data volume: M level –> G level –> T level –> Massive amount is in front of you

Amount of data*Number of users->Relationship number of users: tens of thousands->hundreds of thousands->millions->The Internet is fully open

The pain of relational database

Centralized storage systems have become the bottleneck of I/O access:

1. Traditional architecture database nodes require high-performance hosts and are costly. For multi-node RAC databases, since the communication bandwidth between nodes is usually 1Gb s, and the higher is 10Gb s, this order of magnitude bandwidth makes the concurrent processing capabilities between nodes not fully utilized. At the same time, traditional disk arrays are limited by the processing power of the controller and FC port bandwidth, and the IO throughput is usually only a few hundred MB/s. When the database requires a large number of disk reads and writes, IO consumption takes too long.

2. Small computers equipped with high-end storage are expensive, relatively closed, and have poor scalability: the traditional architecture has poor storage scalability, and when the capacity increases, the performance does not improve accordingly. At the same time, the expansion cost is high and the expansion cycle is long.

3. Complex systems bring complexity to deployment and operation, operation and maintenance, and management: Under the traditional architecture, you need to understand multi-dimensional operation and maintenance knowledge and skills, such as database, U ix, storage, etc., for operation The maintenance personnel requirements are extremely high, and the deployment and operation, operation and maintenance and management are extremely complex.

IT infrastructure development trends

The evolution of IT architecture-chimney to cloud

The two-layer computing-storage architecture has the following flaws:

Centralized storage system

IO access boundary, high expansion cost

Complex deployment and operation and maintenance

Cannot quickly respond to IT staff resource needs

Advantages of cloud-based high-performance and high-availability architecture:

high performance
Elastic, flexible and multi-dimensional dynamic expansion
Unify operation and maintenance management, improve data disaster recovery and protection
It will bring about improvements in IT infrastructure performance and x86 platform CPU performance. In the past 10 years, CPU processing speed has increased by 8-10 times; DRAM processing speed has increased by 7-9 times.

Storage and Network Development

In view of the development of IT infrastructure and the development of IT infrastructure, zData, a distributed storage solution based on x86 servers, using flash memory cards and Infiniband switches, can enable Oracle RAC to achieve high processing power and IO throughput. born.

zData software-defined storage is based on x86 PC servers. While greatly improving system performance and maintaining system scalability and agile response capabilities, the cost is greatly reduced compared with traditional architectures. zData Light Storage distributed storage software can support thousands of nodes, and the storage pool capacity can reach 32PB. Supports storage resource pools composed of different storage media. Under the basic configuration (3 storage nodes), it reaches millions of IOPS and the throughput exceeds 20GB/s.

Implementation method:

1. Users can purchase hardware by themselves or recycle used hardware.

2. Yunhe Enmo provides zData software and services, including performance and capacity planning, zData best practices, database migration and optimization, and overall maintenance.

Proposal value:

1. Standard x86 open architecture, flexible and convenient solution.

2. There is no need to use high-end minicomputers and high-end storage, and construction and expansion are extremely cost-effective.

Product structure:

zData Light Storage storage management software

zData Light Storage is a distributed storage management software independently developed by Yunhe Enmo. It is a set of flexible software-defined storage software that makes full use of internal direct-connected storage based on PC servers to create a virtual and scalable A storage pool that performs significantly better than an external traditional Fiber Channel SAN at exponentially lower cost and complexity. In zData, through zData Light Storage storage management software, high-performance x86 servers can be used as storage nodes, so that the internal disk resources of multiple storage nodes can be integrated to form a virtual SAN storage resource pool. On the database node (computing node) You can use the virtual SAN by installing the zData Light Storage client on it. zData Light Storage supports the RDMA protocol under the Infiniband network, which can significantly reduce CPU usage and significantly reduce IO access latency. It has unparalleled performance advantages. zData Light Storage supports mainstream storage media, including PCIe, NVMe, SAS, SATA and other interfaces. SSD disks and cards as well as traditional mechanical hard drives with SAS and SATA interfaces.

Infiniband Network

zData is based on 40~100Gb Infiniband high-speed network and adopts a fully redundant network architecture. On the one hand, it can provide sufficient bandwidth to carry the data transmission of zData LightStorage storage management software, making its performance significantly better than external traditional FC SAN, and the cost and complexity The performance is doubled; on the other hand, it provides guarantee for high-performance reading and writing of Oracle RAC cluster.

zData intelligent management tool zMan

Based on Yunhe Enmo’s professional knowledge and strong team in the Oracle database field, we developed a humanized management and operation and maintenance tool zMan (zData Manager). zMan provides intelligent and convenient one-click installation, deployment, and management for zData. function, which greatly reduces the cost of management and operation and maintenance and provides users with a simpler, more convenient and friendly operating experience in terms of management functions.

Yunhe Enmo zData software-defined storage has the following characteristics: high performance

Storage nodes and computing nodes are interconnected through the high-speed 40~100Gb Infiniband network, and each connected link can transmit storage data. Through zDataLight Storage storage management software, when using flash memory cards, a single storage node can provide IO throughput of more than 12GB/s, IOPS (disk IO requests per second) can reach more than 400,000, and disk access latency is 0.6 Within ms. The IO throughput on a single computing node reaches more than 12GB/s, IOPS (disk IO requests per second) can reach more than 1 million, and the disk access delay is within 0.6ms. The three key capabilities of storage are more than 10 times that of traditional arrays.

Multi-dimensional dynamic expansion

In the zData software-defined storage architecture, computing nodes and storage nodes can be dynamically expanded according to business and application needs, and capacity and performance increase linearly. Similarly, computing and storage nodes can also be dynamically reduced according to business and application needs to ensure maximum availability of resources

Pooling & cloudification of resources

In the zData software-defined storage architecture, computing resources and storage resources are pooled, managed uniformly, and allocated and expanded on demand.

Security and reliability of architecture and software

In zData software-defined storage, servers, Infiniband switches, Infiniband HCA cards and PCIe flash memory cards are all redundant. Failure of any device or node will not affect the availability of online services and there is no single point of failure. At the same time, zData storage management software automatically manages and maintains storage resources. When a node in the distributed storage network is damaged, or storage nodes are moved or added, zData storage management software will reconstruct intact storage "blocks" on existing or intact nodes. ”, so that each storage “block” still has a mirror, ensuring data security. The reconstruction speed can reach 2TB per hour, and can be operated automatically online without manual intervention and without affecting online business.

Unify operation and maintenance management and improve resource utilization

The entire zData resources are managed uniformly through the zdata management platform and automated operation and maintenance greatly reduces the skill requirements for IT managers.
Yunhe Enmo zData software-defined storage’s unique intelligent caching tool zCache provides the following functions:

• Support writeback and pass-through mode

• Support online startup and shutdown of cache

• Support automatic/manual refresh of dirty data

• Support dynamic control of refresh frequency and speed of dirty blocks

Application scenarios and advantages of zdata software-defined storage:

1. Data warehouse and reporting (OLAP) systems: The databases of this type of system have very high requirements for IO performance and storage capacity expansion, but they do not have high requirements for availability and disaster recovery. This type of system is a secondary important system besides the core system in enterprise applications. If you purchase the most high-end large-capacity centralized storage, the cost is very high. This type of system is the best use case for zData software-defined storage solutions.

2. Online Transaction Processing (OLTP) systems: This type of system has high requirements for real-time performance and concurrency, frequent data operations, and correspondingly high requirements for CPU and IO. Since zData software-defined storage solutions can provide sufficient computing power and significantly reduce IO latency, this type of system is also the best use scenario for zData software-defined storage solutions.

3. Database cloudification and integration: Some large enterprises have a large number of databases and need to integrate many unimportant databases to reduce operation and maintenance complexity and costs. Use the zData software-defined distributed storage solution to build a database cloudization and integration platform to improve the performance of the integrated system and increase the flexibility of the system (facilitate expansion), and use the zMan software-defined storage management tool to make it easy for users to maintain the integrated system. . zData supports cloudification and management of OpenStack and VMware platforms.

4. Database backup and disaster recovery under the

The above is the detailed content of Database cloudification: realizing software-defined storage. For more information, please follow other related articles on the PHP Chinese website!