hbase优化相关参数说明
1、hbase.regionserver.handler.count RegionServers处理远程请求的线程数，如果注重TPS，可以调大，默认10 note1：设得越大，意味着内存开销变大，hbase.client.write.buffer * hbase.regionserver.handler.count，hbase.client.write.buffer默认大小为2M n
1、hbase.regionserver.handler.count
RegionServers处理远程请求的线程数，如果注重TPS，可以调大，默认10 note1：值设得越大，意味着内存开销变大，hbase.client.write.buffer * hbase.regionserver.handler.count，hbase.client.write.buffer默认大小为2M note2: 对于提高write的速度，如果瓶颈在做flush、compact、split的速度，磁盘io跟不上，提高线程数，意义不大。2、hfile.block.cache.size
默认0.25，hfile/StoreFile的最大读缓存空间，所占堆空间比例。 note1:参数设定根据应用场景，如果读比写多，建议调大，读写平衡，建议设成0.3,如果读少于写，建议调小 note2:block.cache.size memstore limits 这些内存加起来不要超过60%。因为剩余的内存还要用来做其他事情。否则容易OOM。3、hbase.regionserver.global.memstore.upperLimit
默认0.4，memstores所占最大堆空间比例，如果达到上限，阻塞更新，强制flush数据4、hbase.regionserver.global.memstore.lowerLimit
默认0.35，menstores达到上限，做flush，知道memstores降到该值，停止flush。5、hbase.hstore.blockingStoreFiles
默认7，如果一个hstore里面storefile超过这个数字（每次memstore做flush时会生成一个hstore），会阻塞相应hregion的更新，知道一个compact压缩过程结束，或者阻塞时间超过hbase.hstore.blockingWaitTime（默认90s） note1：hbase.hstore.compactionThreshold，默认3，如果一个hstore里面的storefile数量超过这个数字，一个压缩任务会启动，将所有的storefile合并成一个。如果数量较多，那么会推迟合并过程，但是再执行时，将会消耗更多时间。 note2：对于持续写的系统，这个参数的设置，是为了compact与flush的速度平衡，如果compact的速度远小于flush的速度，有可能造成 文件io过多，造成too many openfile异常，以及给namenode带来更大的压力。6、hbase.hregion.memstore.flush.size、hbase.hregion.memstore.block.multiplier
默认134217728、2 第一个参数：如果一个memstore大小超过flushsize，则启动flush。后台会有一个线程在周期hbase.server.thread.wakefrequency内，定时检查 第二个参数：如果一个memstore大小超过 该值*flushsize，则阻塞更新。该参数可以平衡，写入速度、flush速度、compact速度、split速度7、hbase.regionserver.checksum.verify
默认false，决定，hbase使用自己的数据校验，而不是hdfs的校验。8、hbase.hregion.max.filesize
默认10G，一个region下，任一列簇的hfiles的大小，超过这个值，该region将split成2个region。 note:如果你的数据量增长的比较快，那么还是建议把这个大小调高，可以调成100G，因为越少的region你的集群越流畅，100G的阈值基本可以避免你的region增长过快，甚至你的region数目会长期不变。当然大region在compaction时也会更加缓慢。几十G的region启动和compaction都非常的慢，如果storefile较多，一个compaction可能会持续几天。9、谈谈region数量设置、以及region split过程
个人观点，如果可以尽早对region进行规划，可以提前预判规划好region的数量，这样可以节省split带来的消耗。 note1：人工进行split 设置hbase.hregion.max.filesize的值为LONG.MAX_VALUE,但是建议设成一个较大的值。预先设计region数量为10，或者更少，然后看数据发展情况。 如果数据较少，可以讲major compact的周期调大。如果数据增长比较快，那么可以调用org.apache.hadoop.hbase.util.RegionSplitter接口，主动进行split。
Hot AI Tools
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Undress AI Tool
Undress images for free
Clothoff.io
AI clothes remover
AI Hentai Generator
Generate AI Hentai for free.
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
C++ function parameter type safety check
Apr 19, 2024 pm 12:00 PM
C++ parameter type safety checking ensures that functions only accept values of expected types through compile-time checks, run-time checks, and static assertions, preventing unexpected behavior and program crashes: Compile-time type checking: The compiler checks type compatibility. Runtime type checking: Use dynamic_cast to check type compatibility, and throw an exception if there is no match. Static assertion: Assert type conditions at compile time.
C++ program optimization: time complexity reduction techniques
Jun 01, 2024 am 11:19 AM
Time complexity measures the execution time of an algorithm relative to the size of the input. Tips for reducing the time complexity of C++ programs include: choosing appropriate containers (such as vector, list) to optimize data storage and management. Utilize efficient algorithms such as quick sort to reduce computation time. Eliminate multiple operations to reduce double counting. Use conditional branches to avoid unnecessary calculations. Optimize linear search by using faster algorithms such as binary search.
How to optimize the startup items of WIN7 system
Mar 26, 2024 pm 06:20 PM
1. Press the key combination (win key + R) on the desktop to open the run window, then enter [regedit] and press Enter to confirm. 2. After opening the Registry Editor, we click to expand [HKEY_CURRENT_USERSoftwareMicrosoftWindowsCurrentVersionExplorer], and then see if there is a Serialize item in the directory. If not, we can right-click Explorer, create a new item, and name it Serialize. 3. Then click Serialize, then right-click the blank space in the right pane, create a new DWORD (32) bit value, and name it Star
Advanced usage of reference parameters and pointer parameters in C++ functions
Apr 21, 2024 am 09:39 AM
Reference parameters in C++ functions (essentially variable aliases, modifying the reference modifies the original variable) and pointer parameters (storing the memory address of the original variable, modifying the variable by dereferencing the pointer) have different usages when passing and modifying variables. Reference parameters are often used to modify original variables (especially large structures) to avoid copy overhead when passed to constructors or assignment operators. Pointer parameters are used to flexibly point to memory locations, implement dynamic data structures, or pass null pointers to represent optional parameters.
Vivox100s parameter configuration revealed: How to optimize processor performance?
Mar 24, 2024 am 10:27 AM
Vivox100s parameter configuration revealed: How to optimize processor performance? In today's era of rapid technological development, smartphones have become an indispensable part of our daily lives. As an important part of a smartphone, the performance optimization of the processor is directly related to the user experience of the mobile phone. As a high-profile smartphone, Vivox100s's parameter configuration has attracted much attention, especially the optimization of processor performance has attracted much attention from users. As the "brain" of the mobile phone, the processor directly affects the running speed of the mobile phone.
How to adjust the parameters of the beauty camera to take better-looking photos. Reference for the best parameters of the beauty camera.
Mar 12, 2024 pm 02:07 PM
How to adjust the parameters of the beauty camera to take better-looking pictures? In the beauty camera app, users can choose filters according to their own needs. If they are users who understand the camera parameters, they can also adjust the camera parameters. The adjusted parameters can help users It is more in line with your own imagination. Different parameters can provide different effects! However, there are not many users who can adjust the parameters. Many friends have no idea about contrast, saturation and sharpening. Naturally, there is no problem if they don’t adjust them. , but you can also apply templates, and you can adjust beautiful parameters without understanding it! Let’s take a look! How to set the best parameters for the beauty camera 1. After you open the camera, there is beauty, body, and makeup. Click Beauty to adjust skin peeling, skin color, face slimming, big eyes, thin nose, and dark circles.
What are some ways to resolve inefficiencies in PHP functions?
May 02, 2024 pm 01:48 PM
Five ways to optimize PHP function efficiency: avoid unnecessary copying of variables. Use references to avoid variable copying. Avoid repeated function calls. Inline simple functions. Optimizing loops using arrays.
How to use parameters in MySQL triggers
Mar 16, 2024 am 09:27 AM
A MySQL trigger is a special program used in the database management system to monitor the operations of a specific table and perform corresponding operations based on predefined conditions. When creating a MySQL trigger, we can use parameters to flexibly pass data and information, making the trigger more versatile and applicable. In MySQL, triggers can trigger and execute corresponding logic before or after the INSERT, UPDATE, and DELETE operations of a specific table. Using parameters can make triggers more flexible and can be passed as needed.
