An in-depth analysis of the MongoDB storage engine (with schematic diagram)

This article will introduce you to the relevant knowledge about mongodb and introduce the storage engine in MongoDB. I hope it will be helpful to you!

A brief review

Last time we talked about the mongodb cluster, which is divided into master-slave cluster and sharded cluster. For the shards in the sharded cluster, here is what is needed Pay attention to the following points, let’s review them together:

• For hot data

certain shard keys (shards A shard key is an index field or composite index field that exists in each document in the collection) will cause all read or write requests to operate on a single data block or shard, which will result in a single sharded server If the load is too heavy, the self-increasing shard key will easily cause writing problems [Recommendation: MongoDB video tutorial]

• For indivisible data blocks

For coarse-grained sharding keys, may result in many documents using the same sharding key

In this case, these documents cannot be split into multiple data blocks, this will limit mongodb's ability to evenly distribute data

• Forquery obstacles

Sharding keys and queries There is no correlation, which will cause poor query performance

For the above points, we must be aware of them. If we encounter similar problems in actual work, we can try to learn to deal with them

Today we will take a brief look at What is the storage engine of mongodb

Storage engine

When it comes to the storage engine of mongodb, we need to know that it is in mongodb 3.0 At that time, the concept of pluggable storage engine was introduced

Now there are mainly these engines:

• WiredTiger storage engine
• inMemory Storage Engine

When the storage engine first came out, the default was to use the MMAPV1 storage engine.

MMAPV1 engine, we can probably know it by looking at the name. He uses mmap to do it, and uses the principle of linux memory mapping

The MMAPV1 engine is not used now because the WiredTiger storage engine is better, for example, compare WiredTiger has the following advantages:

• WiredTiger Better performance in read and write operations

WiredTiger can better utilize the processing capabilities of multi-core systems

• WiredTiger The lock granularity is smaller

  The MMAPV1 engine uses table-level locks. When there are concurrent operations on a single table, the throughput will be affected. Limitations

WiredTiger uses document-level locks, which improves concurrency and throughput

• WiredTiger Better compression

WiredTiger uses prefix compression, which saves memory space consumption compared to MMAPV1

And WiredTiger also provides a compression algorithm, which can greatly reduce the impact on the hard disk. Resource consumption

Writing principle of WiredTiger engine

We can see from the above figure that WiredTiger The principle of writing to disk is also very simple The

• application request comes to mongodb, mongodb handles it, and stores the result in the cache
• When the cache reaches 2 G , or when the 60 s timer expires, the data in the cache will be flushed to the disk.
  Careful xdm will know, then if it is exactly 59 seconds now, more than 1 G, the data in the cache has not yet been synchronized to the disk, and mongodb hangs up abnormally. Then won't mongodb lose data?

We can all think with our fingers, How could the designer of mongodb allow this situation to exist, then there must be a solution, as follows

As shown in the picture above, there is an extra journaling buffer and journal file

• journaling buffer

Buffer that stores mongodb addition, deletion and modification instructions

• journal file

Similar to the transaction log in a relational database

The purpose of introducing Journaling is:

Journaling can enable the mongodb database to quickly recover after an unexpected failure

Journaling log function

Journaling's The logging function looks a bit like the aof persistence in redis. It can only be said that it is similar

In mongodb 2.4, the Journaling logging function is enabled by default , when we start the mongod instance, the service will check whether the data needs to be restored

So there will be no mongodb data loss mentioned above

In addition, we need to know here that journaling’s logging function will write logs when mongodb needs to perform write operations, that is, when adding, deleting, or modifying, which will affect performance

But mongodb will not record the read operation in the cache , so it will not be recorded in the journaling log, so the read operation will have no impact

That’s it for today. What I have learned, if there is any deviation, please correct me

The above is the detailed content of An in-depth analysis of the MongoDB storage engine (with schematic diagram). For more information, please follow other related articles on the PHP Chinese website!