kafka环境搭建与实战(1)安装kafka http://zilongzilong.iteye.com/blog/2267913 kafka环境搭建与实战(2)kafka API实战 http://zilongzilong.iteye.com/blog/2267924
1.maven项目中添加依赖
org.apache.kafka kafka-clients 0.9.0.0
2.spring集成kafka
与spring集成spring-kafka.xml
kafka0:9092,kafka1:9092,kafka2 all 0 16384 1 33554432 org.apache.kafka.common.serialization.StringSerializer org.apache.kafka.common.serialization.StringSerializer com.***.kafka.Partitioner.RandomPartitioner kafka0:9092,kafka1:9092,kafka2:9092 group1 true 1000 30000 org.apache.kafka.common.serialization.StringDeserializer org.apache.kafka.common.serialization.StringDeserializer kafka0:9092,kafka1:9092,kafka2:9092 group2 true 1000 30000 org.apache.kafka.common.serialization.StringDeserializer org.apache.kafka.common.serialization.StringDeserializer
3.producer使用
自己的partion策略类
public class RandomPartitioner implements Partitioner {
@Override
public void configure(Map configs) {
}
@Override
public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) {
Random random = new Random();
List partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();
if (numPartitions > 0) {
return Math.abs(random.nextInt()) % numPartitions;
} else {
return 0;
}
}
@Override
public void close() {
}
}
public class ProducerUtil {
private static Producer producer = SpringContextHolder.getBean("testProducer");
public static void produce(String message) {
producer.send(new ProducerRecord("test",message));
}
}
4.consumer使用
public class KafkaServletContextListener implements ServletContextListener {
@Override
public void contextInitialized(ServletContextEvent sce) {
ExecutorService executor = Executors.newFixedThreadPool(2);
executor.execute(new Runnable() {
@Override
public void run() {
KafkaConsumer consumer = SpringContextHolder.getBean("group1Consumer");
consumer.subscribe(Arrays.asList("test"));
while (true) {
ConsumerRecords records = consumer.poll(100);
for (ConsumerRecord record : records) {
record.key();
record.offset();
record.partition();
record.topic();
record.value();
//TODO
}
}
}
});
executor.execute(new Runnable() {
@Override
public void run() {
KafkaConsumer consumer = SpringContextHolder.getBean("group2Consumer");
consumer.subscribe(Arrays.asList("test"));
while (true) {
ConsumerRecords records = consumer.poll(100);
for (ConsumerRecord record : records) {
record.key();
record.offset();
record.partition();
record.topic();
record.value();
//TODO
}
}
}
});
}
@Override
public void contextDestroyed(ServletContextEvent sce) {
}
}
在web.xml中添加listener
com.***.web.listener.KafkaServletContextListener
5.kafka中遇到的错误
5.1 消费者在消费消息的时候，一直报如下错误：
ILLEGAL_GENERATION occurred while committing offsets for group
在网上找到一篇文章http://comments.gmane.org/gmane.comp.apache.kafka.user/10708，但是按照这个调整了auto.commit.interval.ms和session.timeout.ms，但是还是无济于事。
最后的根本解决办法是，优化消费者的处理逻辑，因为我在消费这种用到了jredis，jredis中对于exist、get时间复杂度为o（1），而smembers方法时间复杂度为o（N），我做的是一是优化代码，二是尽量优先用jredis中对于exist、get方法，然后部署上去，问题经观察，没有出现了。
总之，解决办法是优化消费者代码，减少耗时，让消费者能及时反馈消费状态给zookeeper