Extensions
Grainite is often deployed in environments where other services or products are already running, and in many cases, there is need to either consume data from or push data to those other products. As opposed to building integrations for those products into the core Grainite product, we've introduced Extensions. Extensions are individual packages that contain Tasks, Handlers and other common code needed to integrate with those products.
Grainite Extensions enjoy the same platform guarantees as any Grainite application. They are
- No additional infrastructure requirement. Extensions run directly on Grainite platform eliminating additional resource and operational costs.
- No data loss, duplication, or missed records. Once a record is pulled from the source, it is guaranteed to be processed and written to the sink.
- Source or sink data from/to external components at scale with sub-second latencies.
- High availability with built-in fault tolerance.
- Built-in monitoring and observability
Grainite offers two types of Extensions:
Note that Extensions are not the same as Tasks. Extensions are packages that may contain one or more tasks, one or more handlers, and other common library code that we want to share with users.
Extensions are versioned the same way as the main product, so Grainite version 2315 will also have extensions versioned as 2315. However, Extensions are not automatically installed on the server; they are designed to be integrated and packaged along with client applications.
For Java extensions, we publish Maven packages within our Gitlab Maven repository for customer applications to declare dependencies on and package extensions along with their application packages.
An example of how an extension may be used within a client application is below. In this case, the application wants to read data from the CDC tables for SQLServer and pull them into a Grainite topic. From there, further processing can be done by Grainite handlers. Instead of writing custom code to do so, using the Tasks framework, the pre-built
grainite-sqlserver
extension can be used.app.yaml
app_name: payments_app
package_id: org.samples
jars:
- target/sqlserver-jar-with-dependencies.jar
tasks:
- task_name: sqlserver_cdc
taskClass: ext.grainite.tasks.sqlserver.SQLServerCDCReaderTask
taskInstanceClass: ext.grainite.tasks.sqlserver.SQLServerCDCReaderInstance
config:
connectionUrl: $secret:sqlserver_connectionUrl
database: TestDB
tables: cards;transactions
userid: $secret:sqlserver_userid
password: $secret:sqlserver_password
combine_txn: true
output:
topic: payments_topic
topics:
- topic_name: payments_topic
key_name: id
key_type: string
value_type: json
pom.xml
<dependencies>
...
<dependency>
<groupId>com.grainite</groupId>
<artifactId>libgrainite</artifactId>
<version>2315</version>
</dependency>
<dependency>
<groupId>ext.grainite</groupId>
<artifactId>grainite-sqlserver</artifactId>
<version>2315</version>
</dependency>
...
</dependencies>
The pages within this section provide usage details for each individual extension.
Last modified 2mo ago