spark在yarn-cluster上面执行报错

在单机模式下执行成功的spark程序，在yarn上面就报错。异常信息如下：

 1 14/08/14 02:05:42 INFO DAGScheduler: Completed ResultTask(2, 0)

 2 14/08/14 02:05:42 INFO DAGScheduler: Stage 2 (saveAsTextFile at FileUtil.scala:114) finished in 0.179 s

 3 14/08/14 02:05:42 INFO SparkContext: Job finished: saveAsTextFile at FileUtil.scala:114, took 0.331739293 s

 4 14/08/14 02:05:42 INFO TaskSetManager: Finished TID 2 in 184 ms on localhost (progress: 1/1)

 5 14/08/14 02:05:42 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool 

 6 14/08/14 02:05:42 INFO ApplicationMaster: finishApplicationMaster with SUCCEEDED

 7 Exception in thread "main" java.lang.AssertionError: assertion failed

 8     at scala.Predef$.assert(Predef.scala:165)

 9     at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkContextInitialized(ApplicationMaster.scala:222)

10     at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:111)

11     at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:469)

12     at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:53)

13     at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:52)

14     at java.security.AccessController.doPrivileged(Native Method)

15     at javax.security.auth.Subject.doAs(Subject.java:415)

16     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

17     at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:52)

18     at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:468)

19     at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

20 14/08/14 02:05:44 INFO ApplicationMaster: AppMaster received a signal.

21 14/08/14 02:05:44 INFO ApplicationMaster: Deleting staging directory .sparkStaging/application_1408004797389_0007

从日志上面分析，job执行成功了。但貌似是没有获取到yarn返回的结果信息。

debug后发现是下面的问题：

spark-submit --class org.andy.hadoop.ETL --master yarn-cluster  ../lib/rdbms-0.0.1-SNAPSHOT-jar-with-dependencies.jar /dest/ETL2

job以yarn-cluster形式执行，但代码中初始化的为：

1  var conf = new SparkConf().setAppName("testFilter").setMaster("local")

2  var sc = new SparkContext(conf)

以local的形式初始化的。所以接收不到yarn的返回结果。修改后：

1     var conf = new SparkConf().setAppName("testFilter").setMaster("yarn-cluster")

2     var sc = new SparkContext(conf)

执行成功！