Spark job fails in yarn-cluster mode -

-

my job runs in spark on yarn-client mode , fails on yarn-cluster mode, stating error "file not exist: hdfs://192.xxx.x.x:port/user/hduser/.sparkstaging/application_1442810383301_0016/pyspark.zip". although shows has uploaded file above directory!! cause??

this full error log:

 application application_1449548654695_0003 failed 2 times due container appattempt_1449548654695_0003_000002 exited exitcode: -1000 more detailed output, check application tracking page:http://server1:8088/cluster/app/application_1449548654695_0003then, click on links logs of each attempt. diagnostics: file not exist: hdfs://192.168.0.109:54310/user/hduser/.sparkstaging/application_1449548654695_0003/pyspark.zip java.io.filenotfoundexception: file not exist: hdfs://192.168.0.109:54310/user/hduser/.sparkstaging/application_1449548654695_0003/pyspark.zip @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1309) @ org.apache.hadoop.hdfs.distributedfilesystem$22.docall(distributedfilesystem.java:1301) @ org.apache.hadoop.fs.filesystemlinkresolver.resolve(filesystemlinkresolver.java:81) @ org.apache.hadoop.hdfs.distributedfilesystem.getfilestatus(distributedfilesystem.java:1301) @ org.apache.hadoop.yarn.util.fsdownload.copy(fsdownload.java:253) @ org.apache.hadoop.yarn.util.fsdownload.access$000(fsdownload.java:63) @ org.apache.hadoop.yarn.util.fsdownload$2.run(fsdownload.java:361) @ org.apache.hadoop.yarn.util.fsdownload$2.run(fsdownload.java:359) @ java.security.accesscontroller.doprivileged(native method) @ javax.security.auth.subject.doas(subject.java:415) @ org.apache.hadoop.security.usergroupinformation.doas(usergroupinformation.java:1657) @ org.apache.hadoop.yarn.util.fsdownload.call(fsdownload.java:358) @ org.apache.hadoop.yarn.util.fsdownload.call(fsdownload.java:62) @ java.util.concurrent.futuretask.run(futuretask.java:262) @ java.util.concurrent.executors$runnableadapter.call(executors.java:471) @ java.util.concurrent.futuretask.run(futuretask.java:262) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745) failing attempt. failing application.

could provide full error log?

also did set master of application "yarn-cluster"? can python so:

conf = sparkconf().setappname(appname).setmaster("yarn-cluster") sc = sparkcontext(conf=conf)

Comments

Post a Comment

Popular posts from this blog

java - Date formats difference between yyyy-MM-dd'T'HH:mm:ss and yyyy-MM-dd'T'HH:mm:ssXXX -

-
i trying parse date 2014-12-03t10:05:59.5646+08:00 using these 2 formats: yyyy-mm-dd't'hh:mm:ss yyyy-mm-dd't'hh:mm:ssxxx when parse using yyyy-mm-dd't'hh:mm:ss works fine, when parse yyyy-mm-dd't'hh:mm:ssxxx parseexception thrown. which correct format parse date , difference between these 2 formats? note : cannot use joda :( those valid formats: yyyy-mm-dd't'hh:mm:ss.sssz >>> e.g.: 2001-07-04t12:08:56.235-0700 yyyy-mm-dd't'hh:mm:ss.sssxxx >>> e.g.: 2001-07-04t12:08:56.235-07:00 edit: btw, "x" refer (iso 8601 time zone)
Read more

c# - Get rid of xmlns attribute when adding node to existing xml -

-
i have xml-file: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:d>some content</d> </a> </root> i need add new node in specific place, code is: var doc = xdocument.load(file); xnamespace ns2 = "namespace"; doc.element(ns2 + "root").element(ns2 + "a").element(ns2 + "c").addafterself( new xelement(ns2+"new", new xelement("new1", new xelement("new2","some content"), new xelement("new3", "some content")))); the output is: <ns2:root xmlns:ns2="namespace"> <ns2:a> <ns2:b>some content</b> <ns2:c>some content</c> <ns2:new> <new1 xmlns=""> <new2>some content</new2> ...
Read more