Kafka Extension
The Kafka extension enables reading from and writing to Kafka topics in Grainite applications.
Setup
In order to be able to use the Kafka extension, first include it as a dependency in your application's pom.xml
file.
Replace {GRAINITE-VERSION}
with the version of Grainite you are also using for libgrainite (the Grainite Client library for Java).
Contents
The Kafka Extension includes:
KafkaReaderTask
: Task to continuously read from a given Kafka topic. The task reads the Kafka topic in parallel for each partition in the source topic.KafkaWriterHandler
: Handler that will write it’s input payload to a given (or multiple) Kafka topics
KafkaReaderTask
Note: Only Kafka topics that contain JSON / String payloads are currently supported.
Usage
To include this task in your application, you must specify the taskClass ext.grainite.tasks.kafkareader.KafkaReaderTask
and taskInstanceClass ext.grainite.tasks.kafkareader.KafkaReaderTaskInstance
in your application's configuration YAML file.
Below are the configuration options that can be passed in under config
:
Property | Required? | Value | Description |
---|---|---|---|
| REQUIRED | Example: | Kafka bootstrap server setting (can be found in |
| REQUIRED | Example: | Name of Kafka topic to read from |
| REQUIRED | For String: | Uses the given deserializer provided by Apache Kafka to deserialize the key from the payload. |
| REQUIRED | For String: For Avro: For Protobuf: | Uses the given deserializer provided by Apache Kafka to deserialize the value from the payload. |
| Optional | Example: | URL of schema registry to use look up schema |
| Optional | Example: | Grainite topic to emit output of this task to |
| Optional | Example: | Grainite table to emit output of this task to via Grain message ( |
| Optional | Example: | Given action to invoke via Grain Message to the table specified in |
KafkaWriterHandler
Note: Only Grainite topics that contain JSON payloads are currently supported.
Usage
To include this handler in your application, you must specify the class_name ext.grainite.handlers.KafkaWriterHandler
in your application's configuration YAML file.
Below are the configuration options that can be passed in under config
:
Property | Required? | Value | Description |
---|---|---|---|
| REQUIRED | Example: | Kafka bootstrap server setting (can be found in |
| REQUIRED | Example: | Name of Kafka topic to write to |
| REQUIRED | Example: | Field in the payload that will be used as the Key in the generated event. |
| Optional | Example: | Kafka topic names field in the payload which are used to determine which topic(s) to write the payload to. Kafka topic names should be separated by ';'
If the |
Kafka is a trademark of the Apache Software Foundation
Last updated