Home > Backend Development > PHP Tutorial > Spark pipe + PHP wordcount implementation

Spark pipe + PHP wordcount implementation

WBOY
Release: 2016-07-29 09:00:02
Original
2692 people have browsed it

php’s job is to turn each word into an RDD

<code><span><span><?php</span><span>$in</span> = fopen(<span>'php://stdin'</span>,<span>'r'</span>);
<span>while</span>(!feof(<span>$in</span>))
{
  <span>$temp</span> = explode(<span>" "</span>,fgets(<span>$in</span>));
  <span>for</span> (<span>$i</span>=<span>0</span>;<span>$i</span><count(<span>$temp</span>);<span>$i</span>++){
     printf(<span>"%s\n"</span>,<span>$temp</span>[<span>$i</span>]);  
  } 
}
<span>?></span></span></code>
Copy after login
<code>package test

import org<span>.apache</span><span>.spark</span><span>.SparkConf</span>
import org<span>.apache</span><span>.spark</span><span>.SparkContext</span>object PipeTest {

  def main(args: Array[String]) {
    val sparkConf = new SparkConf()<span>.setAppName</span>(<span>"pipe Test"</span>)
    val sc = new SparkContext(sparkConf)
    val a = sc<span>.textFile</span>(<span>"/home/gt/wordcount.txt"</span>, <span>3</span>)
    val result = a<span>.pipe</span>(<span>"php /home/gt/spark/bin/test.php"</span>)<span>.map</span>(<span>x</span> => (<span>x</span>, <span>1</span>))<span>.reduceByKey</span>(_ + _)
    result<span>.foreach</span> { <span>x</span> => println(<span>"!!!!! "</span> + <span>x</span>) }
    sc<span>.stop</span>()
  }

}</code>
Copy after login
').addClass('pre-numbering').hide(); $(this).addClass('has-numbering').parent().append($numbering); for (i = 1; i ').text(i)); }; $numbering.fadeIn(1700); }); });

The above introduces the wordcount implementation of Spark pipe + PHP, including the relevant aspects. I hope it will be helpful to friends who are interested in PHP tutorials.

Related labels:
source:php.cn
Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
Popular Tutorials
More>
Latest Downloads
More>
Web Effects
Website Source Code
Website Materials
Front End Template