Comment on page
App Configuration Options
YAML file configuration options
The tables and topics for Grainite applications are defined in a YAML configuration file:
Java config
Python config
app.yaml
1
app_name: my_app
2
package_id: org.sample
3
env_name: test
4
# Location of jar file relative to this config file
5
jars:
6
- target/my_app-jar-with-dependencies.jar
7
8
topics:
9
- topic_name: my_topic
10
key_name: key_id
11
key_type: string
12
value_type: json
13
14
exception_handlers:
15
DefaultExceptionHandler:
16
class_name: org.sample.my_app.MyExceptionHandler
17
method_name: handleRuntimeException
18
exceptions:
19
- java.lang.RuntimeException
20
StatusCodeExceptionHandler:
21
status_code: 13
22
reason: User not found.
23
exceptions:
24
- org.sample.my_app.UserNotFound
25
26
imports:
27
- name: DatabaseCDC
28
config:
29
connection: https://database:1234
30
table: my_table
31
32
tables:
33
- table_name: my_table
34
key_type: string
35
value_type: json
36
auto_index_json: true
37
maps:
38
- id: 1
39
name: stats
40
key_type: string
41
value_type: double
42
action_handlers:
43
- name: MyActionHandler
44
type: java
45
class_name: org.sample.my_app.MyActionHandler
46
timeout: 30
47
config:
48
param_one: foo
49
actions:
50
- action_name: handleOrders
51
# `method_name` is optional. Default method_name is the `action_name`
52
method_name: handleOrders
53
# `handles_multiple_actions` is optional. Default is `false`.
54
handles_multiple_actions: false
55
error_sink_topic: handleOrderErrorSink
56
exception_handlers:
57
- DefaultExceptionHandler
58
- StatusCodeExceptionHandler
59
subscriptions:
60
- subscription_name: mySubscription
61
topic_name: my_topic
62
topic_key: key_id
app.py.yaml
1
app_name: my_app
2
python_dir: src
3
env_name: test
4
5
topics:
6
- topic_name: my_topic
7
key_name: key_id
8
key_type: string
9
value_type: json
10
11
tables:
12
- table_name: my_table
13
key_type: string
14
value_type: json
15
auto_index_json: true
16
maps:
17
- id: 1
18
name: stats
19
key_type: string
20
value_type: double
21
action_handlers:
22
- name: MyActionHandler
23
type: python
24
script_path: handlers/MyActionHandler.py
25
timeout: 30
26
config:
27
param_one: foo
28
actions:
29
- action_name: handle_orders
30
# `method_name` is optional. Default method_name is the `action_name`
31
method_name: handle_orders
32
# `handles_multiple_actions` is optional. Default is `false`.
33
handles_multiple_actions: false
34
subscriptions:
35
- subscription_name: mySubscription
36
topic_name: my_topic
37
topic_key: key_id
app_name: my_app
Name of the app to load onto the server. With this, all topics and tables within this app will have their names prepended with the app name. This allows for multiple apps to have the same table and topic names, as well as integrate with each other.
package_id: org.sample
Project-related package ID. This field is used by
gx gen all to generate classes and files. For example, for the YAML above, gx gen all -M would generate the files with the following structure:src
└── main
└── java
└── org
└── sample
└── my_app
├── Client.java
├── Constants.java
└── MyHandler.java
python_dir: src
This is where you specify the location of the Python source code files, if you application uses any Python code. This is used as the path prefix for the
script_path of your Python action handlers (see the Python example in Action Handlers below). For example, you would put src here if your Python client and action handler scripts were stored like the following:src
├── Client.py
└── handlers
└── ActionHandler.py
Then the
script_path of your action handlers defined in ActionHandler.py would be handlers/ActionHandler.py.env_name: test
The name of the environment for this application. This can be overridden for various
gx commands by using --env_name.Applications are scoped by environments, allowing for instances of the same application to run on a single server in complete isolation.
Environment names must not start with a double leading underscore (
__) and must not contain a colon (:).jars:
- target/my_app-jar-with-dependencies.jar
The path, relative to the config file, of the JAR files that contain the action handlers. Generally, all action handlers are packaged within a single JAR, but if that is not the case, it is possible to provide multiple JAR files:
jars:
- target/jar1.jar
- target/jar2.jar
- target/jar3.jar
The JARs provided to Grainite need to be fat/uber JARs, which means that they need to contain all their dependencies. In order to build the fat JAR, you will need to provide additional arguments or options, depending on your build system.
topics:
- topic_name: my_topic
key_name: key_id
key_type: string
value_type: json
A topic has a
topic_name, key_name ,key_type and a value_type.The
topic_name uniquely identifies a topic within the app, the key_name provides a way for grains to subscribe to topics, while key_type and value_typedefine the type of the key and value.There are two kinds of exception handlers:
exception_handlers:
DefaultExceptionHandler:
class_name: org.sample.my_app.MyExceptionHandler
method_name: handleRuntimeException
exceptions:
- java.lang.RuntimeException
An exception handler must have a
method_name and a list of exceptions. Optionally, a class_name can be specified as well. If the class_name is not specified, it defaults to the action handler's parent class.The exception handler name (
DefaultExceptionHandler in the example above) serves as an identifier to allow re-using exception handlers with multiple actions.When an action handler throws a specified exception, the specified "exception handler" is invoked and its result is used.
exception_handlers:
StatusCodeExceptionHandler:
status_code: 13
reason: User not found.
exceptions:
- org.sample.my_app.UserNotFound
A status code exception handler must have a
status_code and a list of exceptions. Optionally, a reason can be specified as well.When an action handler throws a specified exception, the result for the action is automatically converted into an
ActionResult.failure object with the provided status_code and reason.tables:
- table_name: my_table
key_type: string
value_type: long
maps:
- id: 1
name: stats
key_type: string
value_type: double
action_handlers:
...
A table has a
table_name, key_type , value_type maps and action_handlers.The
table_name uniquely identifies a table, the key_type and value_type define the type of key and value for each grain in the table, and maps define the structure for an entry in a map. maps can be identified by either id and/or name . action_handlers contain definitions for action handlers in each grain in the table.This feature was added in 2315.
auto_index_json: true
auto_index_json can be used to have Grainite automatically create secondary indices for all top level properties for a json stored in a Grain’s value.For example, consider this:
Java
Python
String jsonString = """
{
"name": "John",
"location": "Los Angeles",
courses: [
"Computer Science",
"Physics"
]
}
""";
context.setValue(Value.of(jsonString));
json_string = '''
{
"name": "John",
"location": "Los Angeles",
courses: [
"Computer Science",
"Physics"
]
}
'''
context.set_value(Value.of(json_string))
If
auto_index_json: true is set for this Table, Grainite will automatically create indices for name == "John" and location == "Los Angeles" for this Grain. These indices can then be queried using the
table.find() API:Java
Python
Iterator<Grain> iter = table.find("name == 'John' && location == 'Los Angeles'");
// This will print the json from above.
System.out.println(iter.next().getValue().asString());
iter = table.find("name == 'John' && location == 'Los Angeles'")
# This will print the json from above.
print(next(iter).get_value().as_string())
table.find() was added for Python in 2321.Java config
Python config
action_handlers:
- name: MyActionHandler
type: java
class_name: org.sample.my_app.MyActionHandler
timeout: 30
actions:
...
action_handlers:
- name: MyActionHandler
type: python
script_path: handlers/MyActionHandler.py
timeout: 30
actions:
...
An Action Handler has a
name, type, actions and some additional properties based on the provided type.The
name serves as the identifier for the action handler. This field is required if there are more than one action handlers for a table.The
type defines the language that the action handler is written in. Currently, there are three types that are supported:- 1.
java - 2.
python
action_handlers used to be called action_classes in previous versions. Functionally, both are the same and either one of them can be used.However, it is highly recommended to use
action_handlers instead of action_classes as it more accurately defines their roles.Java config
Python config
- type: java
class_name: org.sample.my_app.MyActionHandler
Java action handlers require a
class_name which is the full class name of the defined Java class. These classes should be part of the project package (defined by package_id).- type: python
script_path: handlers/MyActionHandler.py
Python action handlers require a
script_path which is the path to the Python file where this action handler function is defined. The prefix of this path is the application's python_dir , see Python Directory above).timeout was added in 2229 and enables users to configure timeouts (in seconds) for all actions in their action handlers. If an action takes longer than the provided timeout, it is automatically retried. By default, this timeout is set to 60 seconds.actions:
- action_name: myAction
method_name: handleMyAction
error_sink_topic: handleOrderErrorSink
handles_multiple_actions: false
exception_handlers:
- DefaultExceptionHandler
- StatusCodeExceptionHandler
actions define the types of actions that are expected to be handled in the provided action handler. Note that method_name and handles_multiple_actions are both optional.action_name defines the name of the action, while method_name defines the name of the method (in the action handler), that is expected to handle the action. With this, it is possible for a single method to handle multiple actions.If
method_name is not provided for an action, the method_name is the same as the action_name.error_sink_topic defines a topic in which all failed events for this action are sent.exception_handlers define the types of exception handlers associated with this action. When an action throws an exception, the exception handler corrospending to the exception is called and its result is used instead.At the moment,
error_sink_topicand exception_handlers are not available in Python. Only action handlers implemented in Java can make use of these features.There are two types of methods that can be defined within action handlers:
This method takes a single
Action and returns a single ActionResult for a per-action action handler.This method takes a list of
Action and returns a list of ActionResult for a batch-action action handler.handles_multiple_actions can be set to true to use the Multi Action Variant, or false to use the Single Action Variant.By default,
handles_multiple_actions is set to false.subscriptions:
- subscription_name: orderUpdates
topic_name: orders_topic
topic_key: order_id
A method can have multiple subscriptions to topics. A
subscription is defined by a subscription_name, topic_name and topic_key. When a new event is appended to a topic, all subscriptions for that topic are triggered and the method corresponding to the subscription is invoked.The
subscription_name can be used to control the subscription (defer, pause, etc.), while the topic_name and topic_key identify the topic and key (for that topic) for the subscription.To subscribe to a topic in another Application, simply specify the app and topic name, separated by a colon, in the YAML -
<app_name>:<topic_name>.subscriptions:
- subscription_name: ordersSubscription
topic_name: food_app:orders_topic
topic_key: order_id
config:
param_one: foo
config allows users to pass data from the configuration file to action handlers or tasks and access it via GrainContext:Java
Python
String paramOne = context.getConfig().get("param_one");
param_one = context.get_config()["param_one"]
A secret can be passed to the configuration using the prefix
$secret. So for a secret named my_secret which is stored in Grainite, it can be passed in through the configuration file as below:config:
some_token: $secret:my_secret
This can be used by calling the
resolveProperty GrainiteContext API:String token = context.resolveProperty(context.getConfig().get("my_secret"));
Starting in release version 2327, string interpolation for secrets is now supported.
config:
some_token: $secret:{my_secret}-abc
Assuming the value for
my_secret is 123, the above will resolve to 123-abc when context.resolveProperty("some_token") is called in the Action Handler.imports:
- name: DatabaseCDC
config:
connection: https://database:1234
table: my_table
imports provide a convenient way to incorporate complex configurations into your application, enabling Java programs to generate configurations programmatically, based on user-supplied configuration parameters. imports can even be used to share complex application configuration and logic with others through simple configurations.An
import is a way for applications to programmatically provide their required resources (tables, topics, etc.) to gx. Imports are resolved at the time an app is loaded (via
gx load). To view the resolved app config without running gx load, you can use gx beta app-resolve.Each
import config must contain a name which will be used by the gx CLI to search for a properties file - <name>GrainiteImport.properties, in the provided JAR file(s). Consider the example from above; since the name of the import is DatabaseCDC, gx will look for a DatabaseCDCGrainiteImport.properties file in the root of the provided JAR file.The
GrainiteImport.properties must contain a config_provider_class property to point to the class which will generate an application configuration. For instance, the DatabaseCDC properties file could look like this:# DatabaseCDCGrainiteImport.properties
config_provider_class=my.pkg.database_cdc.ConfigProvider
By default,
gx will invoke the generateGrainiteConfig method but this can be changed by providing a config_provider_method property.The method header for the "config generator" method should be the following:
public Map<String, Object> generateGrainiteConfig(Map<String, Object> userConfig)
The method takes in a map representing the
config options defined by the user in their app.yaml. For the DatabaseCDC example, the map will consist of:connection: https://database:1234
table: my_table
The method must return a
Map<String, Object>, representing the resources required by this import. Currently, the following resources can be generated by an import:Here is an example of what
my.pkg.database_cdc.ConfigProvider might generate:public Map<String, Object> generateGrainiteConfig(Map<String, Object> userConfig) {
Map<String, Object> resources = new HashMap<>();
// Declare topics needed by this application.
List<Map<String, Object>> topics = new ArrayList<>();
Map<String, Object> inputTopic = Map.of("topic_name", "cdc_topic", "key_name", "table_id", "key_type", "string");
topics.add(intputTopic);
resources.put("topics", topics);
// Declare tables needed by this application.
List<Map<String, Object>> tables = new ArrayList<>();
Map<String, Object> processorTable = Map.of("table_name", userConfig.get("table"), "key_type", "string", "action_handlers", List.of(...));
tables.add(processorTable);
resources.put("tables", tables);
return resources;
}
The configuration returned will be merged with the existing app.yaml configuration, and will be utilized by
gx load.In the event of conflicting imports, the one declared first in the app.yaml file will take precedence. If an import's configuration clashes with the user-defined configuration specified in the
app.yaml, the user-defined configuration will take precedence.Last modified 4mo ago