Hive(三):之Compression(3)----Sqoop MySQL导入HDFS--使用Snappy压缩

1. 启动HDFS、Yarn

[hadoop@hadoop001 software]$ jps
37985 SecondaryNameNode
40227 Jps
38135 ResourceManager
37818 DataNode
37693 NameNode
38238 NodeManager

2. 查看native库是否安装好

如下,snappy:true,表示支持snappy的压缩

[hadoop@hadoop001 software]$ hadoop checknative
18/12/03 14:06:11 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/12/03 14:06:11 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so

3. 执行导入

sqoop import \
--connect jdbc:mysql://localhost:3306/sqoop \
-m 1 \
--compression-codec Snappy \
--password root \
--username 123456 \
--table emp \
--delete-target-dir \
--target-dir  test/hvie/output/snappy

• 主要参数介绍

参数	介绍
–compression-codec	指定压缩格式 ,如–compression-codec snappy指定使用snappy 压缩
-z	开启压缩,默认gzip

• 注意

  如果不指定压缩(不指定-z 和–compression-codec),但是如果core-site.xmlH和mapred-site.xml里配置了压缩,结果也会根据配置进行压缩

4. 结果

[hadoop@hadoop001 software]$ hadoop fs -ls -R /user/hadoop/test/hvie/output/snappy
-rw-r--r--   1 hadoop supergroup          0 2018-12-03 14:04 /user/hadoop/test/hvie/output/snappy/_SUCCESS
-rw-r--r--   1 hadoop supergroup        513 2018-12-03 14:04 /user/hadoop/test/hvie/output/snappy/part-m-00000.snappy

5. 使用默认压缩

• 执行命令

sqoop import \
--connect jdbc:mysql://localhost:3306/sqoop \
-m 1 \
-z \
--password root \
--username 123456 \
--table emp \
--delete-target-dir \
--target-dir test/hvie/output/snappy-no

• 结果

[hadoop@hadoop001 software]$ hadoop fs  -ls -h test/hvie/output/snappy-no
-rw-r--r--   1 hadoop supergroup          0 2018-12-03 14:42 test/hvie/output/snappy-no/_SUCCESS
-rw-r--r--   1 hadoop supergroup        339 2018-12-03 14:42 test/hvie/output/snappy-no/part-m-00000.gz

• 总结