hadoop SequenceFile
SequenceFile是专为mapreduce设计的，是可分割的二进制格式，以key/value对的形式存储。在存储日志文件时，每一行文本代表一条日志记录。纯文本不合适记录二进制类型的数据。SequenceFile可以作为小文件的容器。
write
先看下 在hadoop中如何写SequenceFile。
private static final String[] DATA = {
"One, two, buckle my shoe",
"Three, four, shut the door",
"Five, six, pick up sticks",
"Seven, eight, lay them straight",
"Nine, ten, a big fat hen"
};
public static void main(String[] args) throws IOException {
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://hadoop:9000");
FileSystem fs = FileSystem.get(configuration);
Path path = new Path("hdfs://hadoop:9000/hadoop/seq/numbers.seq");
IntWritable key = new IntWritable();
Text value = new Text();
SequenceFile.Writer.Option valueOption = SequenceFile.Writer.valueClass(value.getClass());
SequenceFile.Writer.Option keyOption = SequenceFile.Writer.keyClass(key.getClass());
SequenceFile.Writer.Option file = SequenceFile.Writer.file(path);
//指定了 file optiona 就不需要指定 stream
// SequenceFile.Writer.Option stream = SequenceFile.Writer.stream(fs.create(path));
SequenceFile.Writer writer = SequenceFile.createWriter(configuration,file,keyOption,valueOption);
for (int i = 0; i <100; i++) {
key.set(100 -i);
value.set(DATA[i % DATA.length]);
System.out.printf("[%s]\t%s\t%s\n", writer.getLength(), key, value);
writer.append(key, value);
}
IOUtils.closeStream(writer);
}上面的代码基于hadoop 2.6.4，指定了 file option 就不能指定 stream option，否则会抛出异常。原因如下：
// check consistency of options
if ((fileOption == null) == (streamOption == null)) {
throw new IllegalArgumentException("file or stream must be specified");
}read
Configuration configuration = new Configuration();
configuration.set("fs.defaultFS", "hdfs://hadoop:9000");
FileSystem fs = FileSystem.get(configuration);
Path path = new Path("hdfs://hadoop:9000/hadoop/seq/numbers.seq");
SequenceFile.Reader.Option file = SequenceFile.Reader.file(path);
SequenceFile.Reader reader = new SequenceFile.Reader(configuration,file);
Writable key = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(),configuration);
Writable value = (Writable)ReflectionUtils.newInstance(reader.getValueClass(),configuration);
long position = reader.getPosition();
while (reader.next(key,value)){
String syncSeen = reader.syncSeen() ? "*" : "";
System.out.printf("[%s%s]\t%s\t%s\n", position, syncSeen, key, value);
position = reader.getPosition(); // beginning of next record
}
IOUtils.closeStream(reader);输出内容如下：
[128] 100 One, two, buckle my shoe
[173] 99 Three, four, shut the door
[220] 98 Five, six, pick up sticks
[264] 97 Seven, eight, lay them straight
[314] 96 Nine, ten, a big fat hen
[359] 95 One, two, buckle my shoe
[404] 94 Three, four, shut the door
[451] 93 Five, six, pick up sticks
[495] 92 Seven, eight, lay them straight
[545] 91 Nine, ten, a big fat hen
[590] 90 One, two, buckle my shoe
[635] 89 Three, four, shut the door
[682] 88 Five, six, pick up sticks
[726] 87 Seven, eight, lay them straight
[776] 86 Nine, ten, a big fat hen
[821] 85 One, two, buckle my shoe
[866] 84 Three, four, shut the door
[913] 83 Five, six, pick up sticks
[957] 82 Seven, eight, lay them straight
[1007] 81 Nine, ten, a big fat hen
The SequenceFile format
一个SequenceFile是由一个head和 一条或者多条record组成。
SequenceFile头三个字节是SEQ ，用数字表示；其后是version number，如下图，Header也包括其他字段，key/value class，compress等信息。Sync 标志是否允许用户从文件的任何position同步的读取记录。
SequenceFile 的record的格式化依赖于 是否开启压缩，如果压缩，是record compression还是block compression。使用record compression的key不会被压缩。
block 压缩的有点是高效，整个块被压缩，而不是在记录层进行压缩。数据直到达到块的大小时才被压缩，在某一点上整个块被压缩，从而形成整体压缩。默认情况下block的大小与HDFS的block的大小相同，可以使用 io.seqfile.compress.blocksize 设置。
使用压缩，只要在write中添加 一行代码就可以,read 代码可以不变
SequenceFile.Writer.Option compression = SequenceFile.Writer.compression(SequenceFile.CompressionType.BLOCK,new DefaultCodec());