Copyright (c) 2012-2018, Magnus Edenhill.
https://github.com/edenhill/librdkafka
librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.
librdkafka is licensed under the 2-clause BSD license.
For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md
See the wiki for a FAQ.
NOTE: The master branch is actively developed, use latest release for production use.
- High-level producer
- High-level balanced KafkaConsumer (requires broker >= 0.9)
- Simple (legacy) consumer
- Compression: snappy, gzip, lz4, zstd
- SSL support
- SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
- Broker version support: >=0.8 (see Broker version compatibility)
- Stable C & C++ APIs (ABI safety guaranteed for C)
- Statistics metrics
- Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
- RPM package: librdkafka and librdkafka-devel
- Gentoo package: dev-libs/librdkafka
- Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...
- C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
- C++: cppkafka
- D (C-like): librdkafka
- D (C++-like): librdkafkad
- Erlang: erlkaf
- Go: confluent-kafka-go
- Haskell (kafka, conduit, avro, schema registry): hw-kafka
- Lua: luardkafka
- Node.js: node-rdkafka
- OCaml: ocaml-kafka
- PHP: phpkafka
- PHP: php-rdkafka
- Python: confluent-kafka-python
- Python: PyKafka
- Ruby: Hermann
- Ruby: rdkafka-ruby
- Rust: rust-rdkafka
- Tcl: KafkaTcl
- Swift: Perfect-Kafka
- kafkacat - Apache Kafka swiss army knife
- Wikimedia's varnishkafka - Varnish cache web log producer
- Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
- rsyslog
- syslog-ng
- collectd
- logkafka - Collect logs and send to Kafka
- redBorder
- Headweb
- Produban's log2kafka - Web log producer
- fuse_kafka - FUSE file system layer
- node-kafkacat
- OVH - AntiDDOS
- otto.de's trackdrd - Varnish log reader
- Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
- aidp - kafka consumer embedded Lua scripting language in data process framework
- Yandex ClickHouse
- NXLog - Enterprise logging system, Kafka input/output plugin.
- large unnamed financial institutions
- and many more..
- Let me know if you are using librdkafka
The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)
NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer
enables encoding the original size in the compression frame header,
which will speed up the consumer.
Use STATIC_LIB_zstd=/path/to/libzstd.a ./configure --enable-static
to enable static ZStd linking.
MacOSX example:
STATIC_LIB_zstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static
./configure
make
sudo make install
NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.
NOTE: See CMake instructions for experimental CMake build (unsupported).
See examples/rdkafka_example.c for an example producer and consumer.
Link your program with -lrdkafka -lz -lpthread -lrt.
The public APIs are documented in their respective header files:
- The C API is documented in src/rdkafka.h
- The C++ API is documented in src-cpp/rdkafkacpp.h
To generate Doxygen documents for the API, type:
make docs
Configuration properties are documented in CONFIGURATION.md
For a librdkafka introduction, see INTRODUCTION.md
See the examples/sub-directory.
See the tests/sub-directory.
File bug reports, feature requests and questions using GitHub Issues
Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.
Commercial support is available from Edenhill services