hadoop2.6集群搭建

下面是自己在公司的测试服务器上面搭建的HA
#nohup java -jar puv_view.jar >> puv_out.file 2>&1 &

#*/2 * * * * sh /usr/local/puv_jar/exc.sh

卸载centos自带的jdk
yum -y remove java java-1.7.0-openjdk-1.7.0.75-2.5.4.2.el7_0.x86_64
yum -y remove java java-1.7.0-openjdk-headless-1.7.0.75-2.5.4.2.el7_0.x86_64
rpm -qa | grep java
-----------------------------------------------
安装jdk
--------------------------------------------------
/etc/hosts
192.168.1.16 shaobao16
192.168.1.17 shaobao17
192.168.1.18 shaobao18
192.168.1.19 shaobao19

/etc/sysconfig/network
NETWORKING=yes
HOSTNAME=主机名
reboot
-----------------------------------------------------
免密码登陆ssh
(1)ssh-keygen -t r
(2)cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
(3)对每一台机器都要执行
ssh-copy-id -i  shaobao17
ssh-copy-id -i  shaobao18
ssh-copy-id -i  shaobao19
(17,18,19都要对所以机器执行以上部分)
--------------------------------------------------
centos7防火墙
systemctl disable firewalld.service
systemctl stop firewalld.service
----------------------
安装zk
1.ZooKeeper
1.1 zk可以用来保证数据在zk集群之间的数据的事务性一致。
2.如何搭建ZooKeeper服务器集群
2.1 zk服务器集群规模不小于3个节点，要求各服务器之间系统时间要保持一致。
2.2 在hadoop0的/usr/local目录下，解压缩zk....tar.gz，设置环境变量
2.3 在conf目录下，修改文件 vi zoo_sample.cfg  zoo.cfg
2.4 编辑该文件，执行vi zoo.cfg
修改dataDir=/usr/local/zk/data
新增
                    server.0=hadoop0:2888:3888
    server.1=hadoop1:2888:3888
            server.2=hadoop2:2888:3888
2.5 创建文件夹mkdir /usr/local/zk/data
2.6 在data目录下，创建文件myid，值为0
2.7 把zk目录复制到hadoop1和hadoop2中
2.8 把hadoop1中相应的myid的值改为1
    把hadoop2中相应的myid的值改为2
2.9 启动，在三个节点上分别执行命令zkServer.sh start
2.10 检验，在三个节点上分别执行命令zkServer.sh status
-------------------------
搭建Hadoop2.6
1.修改hadoop-env.sh 
  export JAVA_HOME=/usr/local/jdk

2.修改core-site.xml

<configuration>
   <property>
     <name>fs.defaultFS</name>
     <value>hdfs://shaobao</value>
  </property>

  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop-2.6.0/tmp</value>
</property>

<property>
    <name>ha.zookeeper.quorum</name>
    <value>shaobao16:2181,shaobao17:2181,shaobao18:2181</value>
</property>
3.修改hdfs-site.xml
<property>
        <name>dfs.replication</name>
        <value>2</value>
     </property>
     <!--这里是搭建Hadoop联邦-->
  <property>
        <name>dfs.nameservices</name>
        <value>shaobao</value>
   </property>
   <!--配置高可靠-->
   <property>
        <name>dfs.ha.namenodes.shaobao</name>
        <value>shaobao16,shaobao18</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.shaobao.shaobao16</name>
        <value>shaobao16:9000</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.shaobao.shaobao16</name>
        <value>shaobao16:50070</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.shaobao.shaobao18</name>
        <value>shaobao18:9000</value>
    </property>
   <property>
        <name>dfs.namenode.http-address.shaobao.shaobao18</name>
        <value>shaobao18:50070</value>
    </property>

   <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://shaobao16:8485/shaobao</value>
    </property>

    <property>
        <name>dfs.ha.automatic-failover.enabled.shaobao</name>
        <value>true</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.shaobao</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
  <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>/opt/hadoop-2.6.0/tmp/journal</value>
  </property>

  <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>

   <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_rsa</value>
    </property>
   
3.修改mapred-site.xml
   <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
          </property>
         
          <property>
                <name>mapred.child.java.opts</name>
                <value>-Xmx2048m</value>
          </property>
         
4修改slaves
shaobao17
shaobao18
shaobao19

5.描述
   journalnode  shaobao16
   HA shaobao16,shaobao18
   集群名称是shaobao
   slaves shaobao17,shaobao18,shaobao19
  
6.启动journalnode
    sbin/hadoop-daemon.sh start journalnode
7.格式化zk
   bin/hdfs  zkfc  -formatZK
8.格式化,启动namenode
   shaobao16执行
   bin/hdfs  namenode  -format
   sbin/hadoop-daemon.sh  start  namenode
   shaobao18
   bin/hdfs  namenode  -bootstrapStandby
   sbin/hadoop-daemon.sh  start  namenode
9.在win7系统C:\Windows\System32\drivers\etc\hosts添加
192.168.1.16  shaobao16
192.168.1.17  shaobao17
192.168.1.18  shaobao18
192.168.1.19  shaobao19

10.启动zkfc
shaobao16和shaobao18
sbin/hadoop-daemon.sh   start   zkfc

http://shaobao16:50070/dfshealth.html#tab-overview  (shaobao16有standby变成active)
http://shaobao18:50070/dfshealth.html#tab-overview
11修改yarn
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>shaobao16</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
12启动yarn
sbin/start-yarn.sh 
查看  http://shaobao16:8088/ 可以看见resouceMananger的相关配置

hadoop jar hadoop-mapreduce-examples-2.6.0.jar wordcount hdfs://shaobao/sort/a hdfs://shaobao/sort/out3/

当我在此运行2g数据时，rm自动杀死job的线程,报内存溢出。下面对yarn-site.xml进行修改
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>shaobao16</value>
</property>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>


<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>22528</value>
    <discription>每个节点可用内存,单位MB</discription>
  </property>
 
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>3000</value>
    <discription>单个任务可申请最少内存，默认1024MB</discription>
  </property>
 
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>16384</value>
    <discription>单个任务可申请最大内存，默认8192MB</discription>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>3</value>
    <discription>每单位的物理内存总量对应的虚拟内存量，默认是2.1，表示每使用1MB的物理内存，最多可以使用2.1MB的虚拟内存总量</discription>
  </property>