Home > Backend Development > Python Tutorial > What is MapReduce? Learn how MapReduce works in three minutes.

What is MapReduce? Learn how MapReduce works in three minutes.

Tomorin
Release: 2020-07-13 16:22:50
Original
6392 people have browsed it

Python has built-in map() and reduce() functions.

If you have read Google's famous paper "MapReduce: Simplified Data Processing on Large Clusters", you can roughly understand the concept of map/reduce.

Let’s look at the map first. The map() function receives two parameters, one is a function and the other is an Iterable. map applies the passed function to each element of the sequence in turn and returns the result as a new Iterator.

For example, for example, we have a function f(x)=x2, and we want to apply this function to a list [1, 2, 3, 4, 5, 6, 7, 8, 9]. You can use map() to implement it as follows:

What is MapReduce? Learn how MapReduce works in three minutes.

Now, we use Python code to implement it:

>>> def f(x):...     return x * x
...>>> r = map(f, [1, 2, 3, 4, 5, 6, 7, 8, 9])>>> list(r)
[1, 4, 9, 16, 25, 36, 49, 64, 81]
Copy after login

map() Pass in The first parameter is f, which is the function object itself. Since the result r is a Iterator and Iterator is a lazy sequence, we use the list() function to let it calculate the entire sequence. and returns a list.

You may think that you don’t need the map() function. You can also calculate the result by writing a loop:

L = []for n in [1, 2, 3, 4, 5, 6, 7, 8, 9]:
    L.append(f(n))
print(L)
Copy after login

It is indeed possible, but, from the above loop Can you understand the code at a glance "apply f(x) to each element of the list and generate a new list as a result"?

So, map(), as a high-order function, actually abstracts the operation rules. Therefore, we can not only calculate simple f(x)=x2, but also calculate any Complex functions, for example, convert all numbers in this list to strings:

>>> list(map(str, [1, 2, 3, 4, 5, 6, 7, 8, 9]))
['1', '2', '3', '4', '5', '6', '7', '8', '9']
Copy after login

The above is the detailed content of What is MapReduce? Learn how MapReduce works in three minutes.. For more information, please follow other related articles on the PHP Chinese website!

Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template