Table of Contents
基本理论基础
Home Database Mysql Tutorial 互联网百万级应用的大数据处理问题

互联网百万级应用的大数据处理问题

Jun 07, 2016 pm 04:24 PM
internet deal with solving issues data use million

我说的大数据量处理是指同时需要对数据进行检索查询,同时有高并发的增删改操作。记得以前在XX做电力时,几百万条数据,那时一个检索查询可以让你等你分钟。现在我是想探讨下对大数据量的处理,那时我就在想例如腾讯,盛大,动辄数以亿计的帐号,怎么能这么

我说的大数据量处理是指同时需要对数据进行检索查询,同时有高并发的增删改操作。记得以前在XX做电力时,几百万条数据,那时一个检索查询可以让你等你分钟。现在我是想探讨下对大数据量的处理,那时我就在想例如腾讯,盛大,动辄数以亿计的帐号,怎么能这么快呢, 于是找到了互联网现在对数据处理的发展。

对于大数据量处理,如果是互联网处理的话,一般分为下面阶段:

  1. 第一阶段,所有数据都装入一个数据库,当数据量大了肯定就会出现问题,就像刚刚说的查询,于是想办法。
  2. 第二阶段,那时肯定想做缓存机制,确实可以如加上缓存Memcached,但缓存也是治标不治本,数据量太大了也是不行,于是有了下面的方法。
  3. 第三阶段,master-slave模式,进行主从数据库,master提供写,slave进行读,这个适合于有写造成数据库卡的方法,XX那个还是不行,于是——
  4. 第四阶段,垂直分库,这个意义还是不大,对于这种采集数据的,于是——
  5. 第五阶段,进行水平分库,这个不错,记得以前从兴也是按这个分时间水平分库,其实可以分的更细点估计效果更好
  6. 第六阶段,用nosql做了,关于nosql怎么做可以参考google的bigtable

其实本文主要目的也是想探讨nosql对大数据量的处理:

NOSQL就是将写操作在内存中进行,定时或按某一条件将内存中的数据直接写到磁盘上,一定基础上是解决了一些问题:

  1. 高并发读写的需求?
  2. 海量数据访问的需求
  3. 数据库横向扩展性的需求

CAP理论来说,nosql是牺牲了一致性,做到了AP,一致性只是保证了最终一致性。

缺点也很明显:

1. 当机器挂了数据将会丢失,可以考虑共享内存解决。

补充:其实这里可以展开了讲,一种是通过共享内存来实现。

集群内存:根据的是Quorum NRW理论,比如你有N台机子用来集群,每次你进行读写数据时可以至少要同步到X个节点才算成功,所以你每次读数据时只需要读大于N-X个节点就能保持你的正确率,其实就是对数据进行的冗余备份,不过我们存的是内存,相对于直接的磁盘操作,跨网络进行内存操作可以更快。

其实还一种保证数据一致性,就是记录日志,当数据每次写操作内存时都进行日志记录,然后再在内存中进行写操作,至少很多数据库就是这样做的,如redis。

2. 内存的限制,内存有限当写数据操作太大的时候内存也会爆。

解决:Bigtable的做法是通过bloom-filter算法合并掉相同的操作,比如UPDATE A='A' ,update A='B'时可以直接合并了。

基本理论基础

nosql理论基础:内存是新的硬盘,硬盘是新的磁盘

关系型数据库都要实现事务ACID,即:原子性(Atomicity),一致性(Consistency),隔离性(Isolation), 持久性(Durability)。

CAP理论:

  • Consistency 一致性
  • Availability -可用性
  • Partition -容错性

?大多数NoSQL数据库都不支持事务,不支持SQL等,所以还是得保留关系型数据库。现在有人提到用内存数据库, 总体如果是简单业务来说,NOSQL的速度比内存数据库更快,但NOSQL最大缺点,不支持事务,不支持SQL查询等。

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Article

R.E.P.O. Energy Crystals Explained and What They Do (Yellow Crystal)
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. Best Graphic Settings
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
R.E.P.O. How to Fix Audio if You Can't Hear Anyone
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌
WWE 2K25: How To Unlock Everything In MyRise
3 weeks ago By 尊渡假赌尊渡假赌尊渡假赌

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Use ddrescue to recover data on Linux Use ddrescue to recover data on Linux Mar 20, 2024 pm 01:37 PM

DDREASE is a tool for recovering data from file or block devices such as hard drives, SSDs, RAM disks, CDs, DVDs and USB storage devices. It copies data from one block device to another, leaving corrupted data blocks behind and moving only good data blocks. ddreasue is a powerful recovery tool that is fully automated as it does not require any interference during recovery operations. Additionally, thanks to the ddasue map file, it can be stopped and resumed at any time. Other key features of DDREASE are as follows: It does not overwrite recovered data but fills the gaps in case of iterative recovery. However, it can be truncated if the tool is instructed to do so explicitly. Recover data from multiple files or blocks to a single

Open source! Beyond ZoeDepth! DepthFM: Fast and accurate monocular depth estimation! Open source! Beyond ZoeDepth! DepthFM: Fast and accurate monocular depth estimation! Apr 03, 2024 pm 12:04 PM

0.What does this article do? We propose DepthFM: a versatile and fast state-of-the-art generative monocular depth estimation model. In addition to traditional depth estimation tasks, DepthFM also demonstrates state-of-the-art capabilities in downstream tasks such as depth inpainting. DepthFM is efficient and can synthesize depth maps within a few inference steps. Let’s read about this work together ~ 1. Paper information title: DepthFM: FastMonocularDepthEstimationwithFlowMatching Author: MingGui, JohannesS.Fischer, UlrichPrestel, PingchuanMa, Dmytr

The operation process of WIN10 service host occupying too much CPU The operation process of WIN10 service host occupying too much CPU Mar 27, 2024 pm 02:41 PM

1. First, we right-click the blank space of the taskbar and select the [Task Manager] option, or right-click the start logo, and then select the [Task Manager] option. 2. In the opened Task Manager interface, we click the [Services] tab on the far right. 3. In the opened [Service] tab, click the [Open Service] option below. 4. In the [Services] window that opens, right-click the [InternetConnectionSharing(ICS)] service, and then select the [Properties] option. 5. In the properties window that opens, change [Open with] to [Disabled], click [Apply] and then click [OK]. 6. Click the start logo, then click the shutdown button, select [Restart], and complete the computer restart.

Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Google is ecstatic: JAX performance surpasses Pytorch and TensorFlow! It may become the fastest choice for GPU inference training Apr 01, 2024 pm 07:46 PM

The performance of JAX, promoted by Google, has surpassed that of Pytorch and TensorFlow in recent benchmark tests, ranking first in 7 indicators. And the test was not done on the TPU with the best JAX performance. Although among developers, Pytorch is still more popular than Tensorflow. But in the future, perhaps more large models will be trained and run based on the JAX platform. Models Recently, the Keras team benchmarked three backends (TensorFlow, JAX, PyTorch) with the native PyTorch implementation and Keras2 with TensorFlow. First, they select a set of mainstream

Slow Cellular Data Internet Speeds on iPhone: Fixes Slow Cellular Data Internet Speeds on iPhone: Fixes May 03, 2024 pm 09:01 PM

Facing lag, slow mobile data connection on iPhone? Typically, the strength of cellular internet on your phone depends on several factors such as region, cellular network type, roaming type, etc. There are some things you can do to get a faster, more reliable cellular Internet connection. Fix 1 – Force Restart iPhone Sometimes, force restarting your device just resets a lot of things, including the cellular connection. Step 1 – Just press the volume up key once and release. Next, press the Volume Down key and release it again. Step 2 – The next part of the process is to hold the button on the right side. Let the iPhone finish restarting. Enable cellular data and check network speed. Check again Fix 2 – Change data mode While 5G offers better network speeds, it works better when the signal is weaker

The vitality of super intelligence awakens! But with the arrival of self-updating AI, mothers no longer have to worry about data bottlenecks The vitality of super intelligence awakens! But with the arrival of self-updating AI, mothers no longer have to worry about data bottlenecks Apr 29, 2024 pm 06:55 PM

I cry to death. The world is madly building big models. The data on the Internet is not enough. It is not enough at all. The training model looks like "The Hunger Games", and AI researchers around the world are worrying about how to feed these data voracious eaters. This problem is particularly prominent in multi-modal tasks. At a time when nothing could be done, a start-up team from the Department of Renmin University of China used its own new model to become the first in China to make "model-generated data feed itself" a reality. Moreover, it is a two-pronged approach on the understanding side and the generation side. Both sides can generate high-quality, multi-modal new data and provide data feedback to the model itself. What is a model? Awaker 1.0, a large multi-modal model that just appeared on the Zhongguancun Forum. Who is the team? Sophon engine. Founded by Gao Yizhao, a doctoral student at Renmin University’s Hillhouse School of Artificial Intelligence.

The U.S. Air Force showcases its first AI fighter jet with high profile! The minister personally conducted the test drive without interfering during the whole process, and 100,000 lines of code were tested for 21 times. The U.S. Air Force showcases its first AI fighter jet with high profile! The minister personally conducted the test drive without interfering during the whole process, and 100,000 lines of code were tested for 21 times. May 07, 2024 pm 05:00 PM

Recently, the military circle has been overwhelmed by the news: US military fighter jets can now complete fully automatic air combat using AI. Yes, just recently, the US military’s AI fighter jet was made public for the first time and the mystery was unveiled. The full name of this fighter is the Variable Stability Simulator Test Aircraft (VISTA). It was personally flown by the Secretary of the US Air Force to simulate a one-on-one air battle. On May 2, U.S. Air Force Secretary Frank Kendall took off in an X-62AVISTA at Edwards Air Force Base. Note that during the one-hour flight, all flight actions were completed autonomously by AI! Kendall said - "For the past few decades, we have been thinking about the unlimited potential of autonomous air-to-air combat, but it has always seemed out of reach." However now,

The first robot to autonomously complete human tasks appears, with five fingers that are flexible and fast, and large models support virtual space training The first robot to autonomously complete human tasks appears, with five fingers that are flexible and fast, and large models support virtual space training Mar 11, 2024 pm 12:10 PM

This week, FigureAI, a robotics company invested by OpenAI, Microsoft, Bezos, and Nvidia, announced that it has received nearly $700 million in financing and plans to develop a humanoid robot that can walk independently within the next year. And Tesla’s Optimus Prime has repeatedly received good news. No one doubts that this year will be the year when humanoid robots explode. SanctuaryAI, a Canadian-based robotics company, recently released a new humanoid robot, Phoenix. Officials claim that it can complete many tasks autonomously at the same speed as humans. Pheonix, the world's first robot that can autonomously complete tasks at human speeds, can gently grab, move and elegantly place each object to its left and right sides. It can autonomously identify objects

See all articles