记录一下Hive中间和最终结果压缩

阅读更多

中间Lzo,最终Gzip

 

set mapred.output.compress = true;
set mapred.output.compression.codec = org.apache.hadoop.io.compress.GzipCodec;
set mapred.output.compression.type = BLOCK;

set mapred.compress.map.output = true;
set mapred.map.output.compression.codec = org.apache.hadoop.io.compress.LzoCodec;


set hive.exec.compress.output = true;
set hive.exec.compress.intermediate = true;
set hive.intermediate.compression.codec = org.apache.hadoop.io.compress.LzoCodec;

 

 

中间Lzo,最终结果不压缩

 

set mapred.output.compress = true;
set mapred.output.compression.codec = org.apache.hadoop.io.compress.LzoCodec;
set mapred.output.compression.type = BLOCK;

set mapred.compress.map.output = true;
set mapred.map.output.compression.codec = org.apache.hadoop.io.compress.LzoCodec;


set hive.exec.compress.intermediate = true;
set hive.intermediate.compression.codec = org.apache.hadoop.io.compress.LzoCodec;

 

更多大数据Hadoop、Spark、Hive的文章，请关注 我的博客