Hadoop之HDFS

Hadoop之HDFS

HDFS的Shell操作

0. 启动Hadoop集群（方便后续测试）

   [atguigu@hadoop102 ~]$ sbin/start-dfs.sh
   [atguigu@hadoop102 ~]$ sbin/start-yarn.sh
   

1. -help：输出这个命令参数

   [atguigu@hadoop102 ~]$ hadoop fs -help rm
   

2. -ls：显示目录信息

   [atguigu@hadoop102 ~]$ hadoop fs -ls /
   

3. -mkdir：在HDFS上创建目录

   [atguigu@hadoop102 ~]$ hadoop fs -mkdir -p /user/atguigu/input
   [atguigu@hadoop102 ~]$ hadoop fs -mkdir /bigdata0523
   

4. -moveFromLocal：从本地剪切粘贴到HDFS

   [atguigu@hadoop102 hadoop-3.1.3]$ vim qiangge.txt
   ---
   只年说：流年同学你好，可以加个微信吗？
   ---
   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal ./qiangge.txt /bigdata0523
   

5. -appendToFile：追加一个文件到已经存在的文件末尾

   [atguigu@hadoop102 hadoop-3.1.3]$ vim pantongxue.txt
   ---
   流年同学说：你说加就加啊。不给！！
   ---
   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile pantongxue.txt /bigdata0523/qiangge.txt
   

6. -cat：显示文件内容

   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cat /bigdata0523/qiangge.txt
   2025-07-05 12:03:57,587 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
   只年说：流年同学你好，可以加个微信吗？
   流年同学说：你说加就加啊。不给！！
   

7. -chgrp、-chmod、-chown：Linux文件系统中的用法一样，修改文件所属权限

   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -chmod u+x /bigdata0523/qiangge.txt
   

8. -copyFromLocal：从本地文件系统中拷贝文件到HDFS路径去

   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyFromLocal pantongxue.txt /bigdata0523
   

   -put效果和它相同，一般用put就行

   [atguigu@hadoop102 hadoop-3.1.3]$ vim xinge.txt
   ---
   龙哥说：你们俩在干啥，带我一个呗！！！
   ---
   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -put xinge.txt /bigdata0523
   

9. -copyToLocal：从HDFS拷贝到本地

   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -copyToLocal /bigdata0523/qiangge.txt .
   

   -get效果和它相同，一般用get就行

   [atguigu@hadoop102 hadoop-3.1.3]$ vim mengmeng.txt
   ---
   被班主任耽误的舞蹈选手
   ---
   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -moveFromLocal mengmeng.txt /bigdata0523
   [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng
   [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -get /bigdata0523/mengmeng.txt .
   [atguigu@hadoop102 hadoop-3.1.3]$ ll | grep mengmeng
   -rw-r--r--. 1 atguigu atguigu     34 7月   5 12:34 mengmeng.txt
   

10. -cp：从HDFS的一个路径拷贝到HDFS的另一个路径

    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -cp /bigdata0523/mengmeng.txt /
    

11. -mv：在HDFS目录中移动文件

    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /mengmeng.txt /user/atguigu/input
    # 同时还具备改名的功能，下方这种写法 dageoge.txt前方没有明确路径，默认是放到 /user/atguigu 下面了。可以理解为类似于linux也有用户家目录的概念
    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/input/mengmeng.txt dagaoge.txt
    # 移动的同时还可以更改名称
    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -mv /user/atguigu/dagaoge.txt /user/atguigu/input/mengmeng.txt
    

12. -get：等同于copyToLocal，就是从HDFS下载文件到本地

    # 上方copyToLocal已经说过-get，此处不再操作
    hadoop fs -get /bigdata0523/mengmeng.txt .
    

13. -getmerge：合并下载多个文件，比如HDFS的目录/user/atguigu/test 下有多个文件：log.1，log.2，log.3…

    # 不用看后缀名，后缀名没啥用，里面是合并后的内容。getmerge用的比较少
    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -getmerge /bigdata0523 ./teacherAndStudent.avi
    [atguigu@hadoop102 hadoop-3.1.3]$ cat teacherAndStudent.avi 
    被班主任耽误的舞蹈选手
    流年同学说：你说加就加啊。不给！！
    只年说：流年同学你好，可以加个微信吗？
    流年同学说：你说加就加啊。不给！！
    龙哥说：你们俩在干啥，带我一个呗！！！
    

14. -put：等同于copyFromLocal

    # 上方copyFromLocal已经说过-put，此处不再操作
    hadoop fs -put xinge.txt /bigdata0523
    

15. -tail：显示一个文件的末尾

    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail /input/README.txt
    # -n查看指定多少行不能用！实时查看可以跟-f
    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -tail -f /input/README.txt
    

    # 当我们把副本数改为5，再去做操作的时候报错
    [atguigu@hadoop102 hadoop-3.1.3]$ hadoop fs -appendToFile mengmeng.txt /input/README.txt
    2025-07-05 13:11:46,387 WARN hdfs.DataStreamer: DataStreamer Exception
    java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
    	at org.apache.hadoop.hdfs.DataStreamer.findNewDatanode(DataStreamer.java:1304)
    	at org.apache.hadoop.hdfs.DataStreamer.addDatanode2ExistingPipeline(DataStreamer.java:1372)
    	at org.apache.hadoop.hdfs.DataStreamer.handleDatanodeReplacement(DataStreamer.java:1598)
    	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineInternal(DataStreamer.java:1499)
    	at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1481)
    	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:720)
    appendToFile: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[192.168.1.102:9866,DS-da913c4d-3a02-424d-b964-2a3602c1db98,DISK], DatanodeInfoWithStorage[192.168.1.103:9866,DS-3849d948-2fd7-40c0-925b-c498f6c67f76,DISK], DatanodeInfoWithStorage[192.168.1.104:9866,DS-692fc8b7-3c6d-464e-8d81-674708d0ee44,DISK]], original=[DatanodeInfoWithStorage[