Google published three very influential articles in succession from 2003 to 2006, namely GFS released on SOSP in 2003, MapReduce released on OSDI in 2004, and BigTable released on OSDI in 2006. . GFS is related to file systems and has guiding significance for subsequent distributed file system design; MapReduce is a parallel computing programming model used for job scheduling; BigTable is a distributed storage system used to manage structured data. Built on Google technologies such as GFS, Chubby, SSTable, etc. Quite a few Google applications use these three technologies, such as Google Search, Google Earth, and Google Analytics, etc. Therefore, these three technologies are collectively called the "Three Treasures" of Google technology. Today, D Gua Ge is doing his best to do a "cook job" on MapReduce!
Introduction to MapReduce
MapReduce is a programming model and a related implementation of an algorithm model for processing and generating extremely large data sets. The user first creates a
Map function to process a data set based on key/value pairs and outputs the intermediate data set based on key/value pairs; then
creates a Reduce function to merge all data sets with the same intermediate The middle value of the key value.
A picture is worth a thousand words, let’s use a picture to illustrate MapReduce:
Programming Practice
As the saying goes: "Practice brings true knowledge." Whether it's a mule or a horse, you won't know until you take it out for a walk. Therefore, if you really want to understand this principle, it is best to write the code yourself and practice it.
Recently I am learning JavaScript with a few friends, so I pay more attention to JavaScript. Yesterday, when I was browsing around on the Internet, I was surprised to find that there was an awesome person who implemented the MapReduce algorithm using JavaScript. Then I will share it with you, and at the same time add some of my own introduction, hoping to help everyone understand MapReduce. The specific code implementation is as follows:
var Job = {
//Data to be processed
data : [
"We are glad to see you here. This site is dedicated to",
"poetry and to the people who make poetry possible",
" poets and their readers. FamousPoetsAndPoems.com is",
"a free poetry site. On our site you can find a large",
"collection of poems and quotes from over 631 poets",
"Read and Enjoy Poetry",
"I, too, sing America",
"I am the darker brother",
"They send me to eat in the kitchen",
"When company comes" ,
"But I laugh",
"And eat well",
"And grow strong",
"Tomorrow",
"Ill be at the table",
" When company comes",
"Nobodyll dare",
"Say to me",
"Eat in the kitchen",
"Then",
"Besides",
" They'll see how beautiful I am",
"And be ashamed",
"I, too, am America"
],
//Separate each line of strings in the data with spaces Open,
// and "reorganize" it into an object in the format of {key: word, value: 1}, return an object array
map : function(line) {
var splits = line.split(" ");
var temp = [];
for(var i=0; i
temp.push({key : splits[i], value : 1} );
}
return temp;
},
//Calculate the number of times each word appears in "data"
reduce: function(allSteps) {
var result = {};
for(var i=0; ivar step = allSteps[i];
result[step.key] = result[step. key] ? (result[step.key] 1) : 1;
}
return result;
},
//Initialization, and also the entry point for operation.
init : function() {
var allSteps = [];
for(var i=0; i//If multiple threads can be called here The Job.map function is even more realistic. ? ?
allSteps = allSteps.concat(Job.map(Job.data[i]));
}
//The only drawback is that the Job.reduce function cannot be called by multiple threads here? ?
var result = Job.reduce(allSteps)
console.log(JSON.stringify(result));
}
}; // Job
//Start executing
Job .init();
Copy these codes and paste them directly into the browser's console (Console), or put them into an HTML file and open it with a browser. You can see the following effect in the console output:
Flaw in the ointment After this article was published, some netizens "roared": "What kind of MapReduce can a js that doesn't even have multi-threading do?" In fact, Brother D Gua also asked this question Found it. After seeing the explanation of this code, Brother D Gua wondered if JavaScript was not a single process? How can we still simulate MapReduce? After carefully reading the code and single-step debugging, D Gua Ge's views are further confirmed. (The questions about D Gua Ge have been commented in the code.)
However, if you think about it again, these do not affect our understanding of the principles of MapReduce. This is just a single process, the most basic version. After understanding this first, it may be easier to understand the whole multi-threading process.
To be continued In fact, Brother D is now considering using Java to implement a multi-threaded version based on this example, so that the simulated MapReduce will be more realistic. After Brother D Gua has thought through some issues clearly, he will send out the code. Stay tuned!
![[Web front-end] Node.js quick start](https://img.php.cn/upload/course/000/000/067/662b5d34ba7c0227.png)
