Hadoop Exception 以及解决办法

阅读更多

1.....今天写 Hadoop程序的时候 出现了这么一个问题

java.lang.RuntimeException: java.io.EOFException
	at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:165)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1268)
	at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:35)
	at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:88)
	at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1600)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2019)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
	at java.io.DataInputStream.readInt(Unknown Source)
	at com.mjm.tq.Weather.readFields(Weather.java:55)
	at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:158)
	... 16 more

 

检查 错误原因：

	@Override
	public void readFields(DataInput in) throws IOException {
		this.year = in.readInt();
		this.month = in.readInt();
		this.day = in.readInt();
		this.temperature = in.readInt();
	}
	@Override
	public void write(DataOutput out) throws IOException {
                //这边应该是writeInt()
		out.write(year);
		out.write(month);
		out.write(day);
		out.write(temperature);
	}

 

2...z在本地从测试环境下，在windows  Eclipse上运行hadoop程序时

总是报如下错误：

Diagnostics: Exception from container-launch.
Container id: container_1492698686128_0006_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control

Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
	at org.apache.hadoop.util.Shell.run(Shell.java:479)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2017-04-21 09:44:15,747 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0

 

找出 错误原因 是因为

项目中的classpath下不能有  服务器上的配置文件

至于为什么  就有点 不太懂了，可以看看网上其他的blog 

参考：http://www.aboutyun.com/thread-8498-1-1.html

 

 

3.对了 我还遇到过一个问题   环境：服务器模式

就是MR程序状态一直在  JobRunnng状态  卡在这边 >=30min,后来在网上 看看有没有解决办法

 

在yarn-site.xml中加了:

      

    yarn.nodemanager.resource.memory-mb
    20480


   yarn.scheduler.minimum-allocation-mb
   2048


    yarn.nodemanager.vmem-pmem-ratio
    2.1

 

加了 以上这几句 就好了，可以正常运行

  出现此故障的原因应该是，在每个Docker分配的内存和CPU资源太少，不能满足Hadoop和Hive运行所需的默认资源需求。

参考：http://blog.csdn.net/dai451954706/article/details/50464036

 

 

• 
• 大小: 24.7 KB

• 查看图片附件