App Configuration

LibEMP Wiring Guide

• NOTE: This configuration guide is not complete, nor implemented. It is a possibility which meets the neccessary requirements for a LibEMP Configuration guide. See Notes on a Wiring Language, for more details.

LibEMP has a number of constructs that need to be wired together. On the application side Monitors and Sinks need to be linked up to opposite sides of a Buffer. On the Processing side, Sinks can be stacked on one another to build a fault-tolerant chain of processing. Sinks can also be given unique Fault-Handling functions when built into these stacks, thereby customizing the processing behaviour.

Buffers

Buffers store events either locally or not. A Node must have at least one buffer, and multiple LibEMP applications can choose to use that buffer or not. I would recommend only defining the buffer as part of the Application wiring if your application has a specific Buffer implementation that it requires.

{buffer, Module, [Configs]}.
{buffer, Name, Module, [COnfigs]}.

Sinks

We define named configurations for Event Sinks as follows. They can be given unique names, but default to the module's name.

{sink, Module, [Configs]}.
{sink, Name, Module, [Configs]}.
{sink, Buffer, Name, Module, [Configs]}.

Stack Sink

LibEMP has a special type of Sink, called the Stack Sink. The Stack defines the order a sink should be allowed to process an event, as well as how to handle failures at any point along its processing. Stacks can also have sub-stacks within them.

{stack, [ StackItem, ... ]}.
{stack, Buffer, [ StackItem, ... ]}.
{stack, DefaultHandler, Buffer, [ StackItem, ... ]}.

Stack Items can be a named Sink, or a stack definition internally:

StackItem := SinkName :: atom() 
           | { SinkName, FaultHandler :: atom() | fun( (_,_,_) -> ... ) }
           | {stack, [ StackItem ]}
           | {stack, DefaultFaultHandler, [ StackItem ]}

Fault Handling Functions

Functions can be defined in two different ways (via refrencing it or defining it there).

{fault_function, myhandlername, fun mymod:fault_handler_function/3}.
{fault_function, myhandlername, fun( RetryCount, Event, ErrorReason ) -> ... end}.

You can name it via the fault_function triple, or inline in the Stack definition.

{stack, fun(_,_,_) -> ... end, _, [...]}.
{stack, [ {mysink, myhandlername}, ...]}.
{stack, [ {mysink, fun mymod:myfun/3}, ... ]}.
{stack, [ {mysink, fun(_,_,_) -> ... end}, ... ]}.

Monitors

By default Monitors are linked to the first Buffer created, but you can force wire them up to a named buffer.

{monitor, Module, [Configs]}.
{monitor, Buffer, Module, [Configs]}.

Includes

You can break out configs into multiple files and have Node configurations be just a set of "include"s. For example:

{include, RelativeOrAbsolutePathToWireFile}.

Examples:

Given our libemp_examples package, we can start up multiple applications all with the same node. We define a node by giving it a buffer and an application wiring. In so far as we can ask the application to parse multiple wire files:

> libemp:start(["config/mylocalnode.wire", "config/cron.wire", "config/weather.wire"]).

Note one can start processing nodes by just starting sinks/stacks, whereas one can build a producer node by just starting monitors. They should both share a buffer though.

Node Examples:

Local Node

% config/mylocalnode.wire
% We just use the simple buffer, which does not require any special features
{buffer, default, libemp_simple_buffer, []}.

Remote Nodes

% config/myremotenode.wire
{buffer,default, libemp_sqs_remote_buffer, [
  {aws_access_id, "AKIAIOSFODNN7EXAMPLE"},
  {aws_access_key, "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"},
  {region, "us-west-1"},
  {queue_name, "libemp_example"}
]}.

Application Examples:

Cron

% config/cron.wire - 
% All Monitors are wired up to the default buffer:
{monitor, libemp_timer_monitor, []}.
%% Note the following can be moved into a new file, 
%% to become a processing node.
% Example Sinks
{sink, cron_sink, []}.
% Stacks:
{stack, [cron_sink]}.

Weather

% config/weather.wire