MXnet调用Kafka方法

好习惯：结论放最前

CPP+CPP

---

Python+Java

kafka的文档只有Java的API，因此一开始考虑在MXnet的Python API跨语言调用Kafka的Java API。大致思路是Python下运行Java的虚拟机。

参考：https://www.cnblogs.com/mumuli/p/5806963.html

Python+Python

后来找到第三方做Kafka的Python API的库，根据更新情况和文档情况，选择了Kafka-python。
根据文档在virtualenv中安装pip install kafka-python（之前已经安装MXNet）。
写了个test文件

from kafka import KafkaProducer
from kafka.client import KafkaClient
if __name__ == "__main__":
    topic = KafkaClient().add_topic("test")
    producer = KafkaProducer(bootstrap_servers='localhost:9092')
    for _ in range(100):
        producer.send('test', b'this is a message')

根据之前的介绍的单机启动部分，启动本地kafka，并开启了一个consumer观察。
测试成功：

image.png

CPP+CPP

librdkafka的代码
基于librdkafka的high level wrapper API cppkafka代码

---

centos:
yum install libgsasl-devel.x86_64
yum install openssl-devel
yum install zlib zlib-devel

---

git clone [email protected]:edenhill/librdkafka.git
mkdir librdkafka-build
cd librdkafka
./configure --prefix=../librdkafka-build
make
sudo make install

# install protobuf: https://github.com/google/protobuf/blob/master/src/README.md

# compile the cppkafka: cmake and boost are needed
git clone [email protected]:mfontanini/cppkafka.git
mkdir cppkafka-build
cd cppkafka-build
# .so :
# cmake ../cppkafka -DRDKAFKA_ROOT_DIR=../librdkafka-build
# .a :
cmake ../cppkafka -DCPPKAFKA_BUILD_SHARED=0 -DRDKAFKA_ROOT_DIR=../librdkafka-build
make
make install

-- Install configuration: ""
-- Installing: /usr/local/lib/libcppkafka.a
-- Installing: /usr/local/include/cppkafka/message_builder.h
-- Installing: /usr/local/include/cppkafka/configuration_base.h
-- Installing: /usr/local/include/cppkafka/producer.h
-- Installing: /usr/local/include/cppkafka/error.h
-- Installing: /usr/local/include/cppkafka/configuration_option.h
-- Installing: /usr/local/include/cppkafka/topic.h
-- Installing: /usr/local/include/cppkafka/message.h
-- Installing: /usr/local/include/cppkafka/metadata.h
-- Installing: /usr/local/include/cppkafka/clonable_ptr.h
-- Installing: /usr/local/include/cppkafka/kafka_handle_base.h
-- Installing: /usr/local/include/cppkafka/configuration.h
-- Installing: /usr/local/include/cppkafka/group_information.h
-- Installing: /usr/local/include/cppkafka/topic_partition_list.h
-- Installing: /usr/local/include/cppkafka/topic_partition.h
-- Installing: /usr/local/include/cppkafka/buffer.h
-- Installing: /usr/local/include/cppkafka/exceptions.h
-- Installing: /usr/local/include/cppkafka/topic_configuration.h
-- Installing: /usr/local/include/cppkafka/macros.h
-- Installing: /usr/local/include/cppkafka/consumer.h
-- Installing: /usr/local/include/cppkafka/utils/consumer_dispatcher.h
-- Installing: /usr/local/include/cppkafka/utils/backoff_performer.h
-- Installing: /usr/local/include/cppkafka/utils/compacted_topic_processor.h
-- Installing: /usr/local/include/cppkafka/utils/backoff_committer.h
-- Installing: /usr/local/include/cppkafka/utils/buffered_producer.h

CPP+Java

JNI