Home Database Mysql Tutorial HBase MVCC and built-in Atomic Operations

HBase MVCC and built-in Atomic Operations

Jun 07, 2016 pm 04:28 PM
and atomic hbase mvcc

By Lars Hofhansl (This is a follow to my ACID in HBase post from March this year) HBase has a few special atomic operations: checkAndPut, checkAndDelete - these simply check a value of a column as a precondition and then apply the Put or D

By Lars Hofhansl

(This is a follow to my ACID in HBase post from March this year)
HBase has a few special atomic operations:
  • checkAndPut, checkAndDelete - these simply check a value of a column as a precondition and then apply the Put or Delete if the check succeeded.
  • Increment, Append - these allow an atomic add to a column value interpreted as an integer, or append to the end of a column, resp.
checkAndPut and checkAndDelete are idempotent in the sense that they can safely be applied multiple time (but note that their outcome might differ when other changes are applied between the retries).

Increment and Append are not idempotent. They are the only non-repeatable operations in HBase. Increment and Append are also the only operations for which the snapshot isolation provided by HBase's MVCC model is not sufficient... More on that later.

In turns out that checkAndPut and checkAndDelete are not as atomic as expected (the issue was raised by Gregory and despite me not believing it first he is right - see HBASE-7051).

A look at the code makes this quite obvious:
Some of the Put optimizations (HBASE-4528) allow releasing the rowlock before the changes are sync'ed to the WAL. This also requires the lock to be released before the MVCC changes are committed so that changes are not visible to other transaction before they are guaranteed to be durable.
Another operation (such as checkAndXXX) that acquires the rowlock to make atomic changes may in fact not see current picture of things despite holding the rowlock as there could be still outstanding MVCC changes that only become visible after the row lock was release and re-acquired. So it might operate on stale data. Holding the rowlock is no longer good enough after HBASE-4528.

Increment and Append have the same issue.

The fix for this part is relatively simple: We need a "MVCC barrier" of sorts. Instead of completing a single MVCC transaction at the end of the update phase (which will wait for all prior transactions to finish), we just wait a little earlier instead for prior transactions to finish before we start the check or get phase of the atomic operation. This only reduces concurrency slightly, since before the end of the operation we have to await all prior transactions anyway. HBASE-7051 does exactly that for the checkAndXXX operations.

In addition - as mentioned above - Increment and Append have another issue, they need to be serializable transactions. Snapshot isolation is not good enough.
For example: If you start with 0 and issue an increment of 1 and another increment of 2 the outcome must always be 3. If both could start with the same start value (a snapshot) the outcome could 1 or 2 depending on which one finishes first.

Increment and Append currently skirt the issue with an ugly "hack": When they write their changes into the memstore they set the memstoreTS of all new KeyValues to 0! The effect is that they are made visible to other transactions immediately, violating HBase's MVCC. Again see ACID in HBase for an explanation of the memstoreTS.
This guarantees the correct outcome of concurrent Increment and Append operations, but the visibility to concurrent scanners is not what you expect. An Incremented and Appended value even for partial rows can be become visible to any scanner at any time even though the scanner should see an earlier snapshot of the data.
Increment and Append are also designed for very high throughput so they actually manipulate HBase's memstore to remove older versions of the columns just modified. Thus you lose the version history of the changes in exchange for avoiding a memstore exploding with version of the many Increments or Appends. This is called "upsert" in HBase. Upsert is nice in that it prevents the memstore being filled will a lot of old value if nobody cares for them. The downside is that is a special operation on the memstore, and hard to get right w.r.t. MVCC. It also does not work with mslab (see this Cloudera blog post for explanation of mslab).

If you don't care about visibility this is a simple problem, since you can just look through the memstore and remove old values. If you care about MVCC, though, you have to prove first that is safe to remove a KV.

I tried to fix this almost exactly a year ago (HBASE-4583), but after some discussions with my fellow committers we collectively gave up on that.

A few days ago I reopened HBASE-4583 and started with a radical patch that gets rid of all upsert-type logic (which set the memstoreTS to 0) and just awaits prior transactions before commencing the Increment/Append. Then I rely on my changes from HBASE-4241 to only flush the versions of columns needed when it is time to flush the memstore to disk. Turns out this is still quite a bit slower (10-15%), since it needs to flush the memstore frequently even thought it leads to mostly empty files. Still that was nice try, as it gets rid of a lot of special code and turns Increment and Append into normal HBase citizens.

A 2nd less radical version makes upsert MVCC aware.

That is actually not as easy as it looks. In order to remove a version of a column (a KeyValue) from the memstore you have to prove that is not and will not be seen by any concurrent or future scanner. That means we have to find the earliest readpoint of any scanner and ensure that there is at least one version of the KV older than that smallest readpoint; then we can safely remove any older versions of that KV from the memstore - because any scanner is guaranteed to see a newer version of the KV.
The "less radical" patch in  does exactly that.

In the end the patch I ended up committed with HBASE-4583 does both:
If the column family that has the column to be incremented or appended to has VERSIONS set to 1, we perform an MVCC aware upsert added by the patch. If VERSIONS is > 1, we use the usual logic to add a KeyValue to the memstore. So now this behaves as expected in all cases. If multiple versions are requested they are retained and time range queries will work even with Increment and Append; and it also keeps the performance characteristics (mostly) when VERSIONS is set to 1.

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
2 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Repo: How To Revive Teammates
4 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
Hello Kitty Island Adventure: How To Get Giant Seeds
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

In-depth analysis of MySQL MVCC principle and implementation In-depth analysis of MySQL MVCC principle and implementation Sep 09, 2023 pm 08:07 PM

An in-depth analysis of the principles and implementation of MySQLMVCC. MySQL is one of the most popular relational database management systems currently. It provides a multiversion concurrency control (MultiversionConcurrencyControl, MVCC) mechanism to support efficient concurrent processing. MVCC is a method of handling concurrent transactions in the database that can provide high concurrency and isolation. This article will provide an in-depth analysis of the principles and implementation of MySQLMVCC, and illustrate it with code examples. 1. M

Using Hadoop and HBase in Beego for big data storage and querying Using Hadoop and HBase in Beego for big data storage and querying Jun 22, 2023 am 10:21 AM

With the advent of the big data era, data processing and storage have become more and more important, and how to efficiently manage and analyze large amounts of data has become a challenge for enterprises. Hadoop and HBase, two projects of the Apache Foundation, provide a solution for big data storage and analysis. This article will introduce how to use Hadoop and HBase in Beego for big data storage and query. 1. Introduction to Hadoop and HBase Hadoop is an open source distributed storage and computing system that can

In-depth interpretation of MySQL MVCC principles and best practices In-depth interpretation of MySQL MVCC principles and best practices Sep 09, 2023 am 11:40 AM

In-depth interpretation of MySQLMVCC principles and best practices 1. Overview MySQL is one of the most widely used relational database management systems. It supports the Multi-Version Concurrency Control (MVCC) mechanism to deal with concurrent access issues. This article will provide an in-depth explanation of the principles of MySQLMVCC and give some best practice examples. 2. MVCC principle version number MVCC is by adding additional

Understand the principles of MySQL MVCC and optimize database transaction processing Understand the principles of MySQL MVCC and optimize database transaction processing Sep 09, 2023 am 09:42 AM

Understand the principles of MySQLMVCC and optimize database transaction processing. In recent years, with the continuous growth of data volume and the improvement of application requirements, the performance optimization of database transaction processing has become a very important link in database development and operation and maintenance. As one of the most commonly used open source relational databases, MySQL's MVCC (Multi-VersionConcurrencyControl) principle plays an important role in transaction processing. This article will introduce the principles of MySQLMVCC and

How to use AND operator and OR operator in SQL statement How to use AND operator and OR operator in SQL statement May 28, 2023 pm 04:34 PM

SQLAND&OR OperatorThe AND and OR operators are used to filter records based on more than one condition. AND and OR combine two or more conditions in the WHERE substatement. The AND operator displays a record if both the first and second conditions are true. The OR operator displays a record if either the first condition or the second condition is true. "Persons" table: LastNameFirstNameAddressCityAdamsJohnOxfordStreetLondonBushGeorgeFifthAvenueNewYorkCarter

What are the knowledge points of Mysql MVCC multi-version concurrency control? What are the knowledge points of Mysql MVCC multi-version concurrency control? May 27, 2023 pm 11:31 PM

1. MVCCMVCC, the full name is Multi-VersionConcurrencyControl, which is multi-version concurrency control. MVCC is a concurrency control method. It is generally used in database management systems to achieve concurrent access to the database and to implement transactional memory in programming languages. The implementation of MVCC in MySQLInnoDB is mainly to improve database concurrency performance and use a better way to handle read and write conflicts, so that even if there are > read and write conflicts, no locking and non-blocking concurrent reading can be achieved. 2. The current reading is like selectlockinsharemode (shared lock), selectforupdate; update, insert, de

How to integrate hbase in springboot How to integrate hbase in springboot May 30, 2023 pm 04:31 PM

Dependency: org.springframework.dataspring-data-hadoop-hbase2.5.0.RELEASEorg.apache.hbasehbase-client1.1.2org.springframework.dataspring-data-hadoop2.5.0.RELEASE The official way to add configuration is through xml, which is simple After rewriting, it is as follows: @ConfigurationpublicclassHBaseConfiguration{@Value("${hbase.zooke

Master the principles of MySQL MVCC and improve data reading efficiency Master the principles of MySQL MVCC and improve data reading efficiency Sep 10, 2023 pm 04:34 PM

Master the principles of MySQLMVCC and improve data reading efficiency Introduction: MySQL is a commonly used relational database management system, and MVCC (Multi-VersionConcurrencyControl) is a commonly used concurrency control mechanism in MySQL. Mastering the MVCC principle can help us better understand the internal working principles of MySQL and improve the efficiency of data reading. This article will introduce the principle of MVCC and how to use this principle to improve data reading efficiency.

See all articles