python - pyspark执行reduce时报错:port out of range:1885434979
PHP中文网
PHP中文网 2017-04-17 16:59:25
0
0
1014

刚装完spark-1.6.0-bin-hadoop2.6.tgz,
Java 1.8.0_65
Python 2.7

运行pyspark:

dd = sc.parallelize([1,2,3,4,5])
dd.reduce(lambda a, b:a+b)

报错:

16/01/14 18:35:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2102 bytes)
16/01/14 18:35:28 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
16/01/14 18:35:28 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/01/14 18:35:28 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
16/01/14 18:35:28 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/01/14 18:35:28 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID 3)
java.lang.IllegalArgumentException: port out of range:1885434979
        at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
        at java.net.Socket.<init>(Socket.java:244)

尝试了其他方式,map可以执行,reduce/collection等都是报这个错。

“port out of range:1885434979” 字面上是端口数太大,没有找到任何设置的地方。

求解。

PHP中文网
PHP中文网

认证高级PHP讲师

全部回复(0)
热门教程
更多>
最新下载
更多>
网站特效
网站源码
网站素材
前端模板
关于我们 免责声明 Sitemap
PHP中文网:公益在线PHP培训,帮助PHP学习者快速成长!