Storyboard Architecture

Storyboard is designed to be used to develop and run fullscreen application user interfaces. Developing a Storyboard based application is straightforward and easy, but relies on understanding a few concepts and terms and how they relate to one another within the Storyboard framework.

[Note]Note

Storyboard Designer and Engine use the same terminology in general, but in some instances, namely rendering properties, Designer uses more human readable names.

Graphical Composition Elements

A Storyboard application is be composed of multiple screens however only a single screen is ever visible at one time.

Each screen is composed of one or more layers. The screen controls how multiple layers have their content composited together to form final display output.  When possible, layers may be mapped directly to graphic hardware layers. A single layer can be referenced by multiple screens, in this case the screen content is shared and not duplicated. When an instance of a layer is used on a screen it is given a position, size, transparency/alpha value and z-order.

[Note]Note

A layer that is associated with a screen is referred to as a layer instance to differentiate it from the generic layer with shared content.

  When a screen is painted the z-order is used to determine which areas of a layer are visible to the user.

Layers contain controls.   Controls are clipped containers for renderable content.  A control is sized and positioned relative to its parent layer and contains a list of zero or more render extensions that describe the type of rendering to occur when a control is damaged and redrawn. Controls also represent the focusable objects within a Storyboard application.

The term model element is used within this document to generically refer to an application, screen, layer, layer instance or control object. All model elements can be associated with zero or more actions or data variables.

Screens

A screen represents the users view at any given moment.  A screen is composed of one or more layers.  These layers are composited to the screen in a z-order (back to front) in order to combine to form the desired output.  Each screen in the system must have a unique name and by the use of actions and events a user can traverse the user interfaces many screens.

Here is an example of a screen containing three layers containing controls with images and text that combine to form the final screen:

Layers

Controls are contained in layers.  A user interface can have 1 or more layers which in turn are composed of 1 or more controls.  Layers are logical groupings of controls and can also react to events and execute actions.  As with controls, layers can be shown or hidden as required.  A single layer can be reused across multiple screens to allow common controls to be displayed.

Controls

A user interface is made up of many controls.  A control is a rectangular area of the screen which can render content and react to events.  Each control can be made up of many render extensions and react to any number of events.  Controls can be shown or hidden and active/inactive as the system requires.  A special type of control called a table is available in order to create a row/column based layout of information.  Each column in a table can have a distinct template for rendering.  This allows the efficient and easy creation of list style information.

Render Extensions

A render extension is the most basic piece of the user interface. It is responsible for rendering a specific type of content.  Render extensions must be bound to a specific control and are not permitted to draw beyond the boundaries of the control to which they are bound.  It is possible to include multiple render extensions on a control and the order in which they are declared in the deployment bundle will define the order in which they perform their rendering.

Some render extensions included with the standard Storyboard release include:

  • Render an image (PNG, GIF, JPEG)

  • Render a text string

  • Render a filled rectangle or polygon

When designing a user interface, it is often possible to achieve the same look and feel by using multiple render extensions on a control or using multiple controls with overlapping behavior.  The general rule of thumb is that render extensions should be used when there is a common event to action binding that would occur for all of the visual components being displayed.  

Events and Actions

Events

Events are the basic communication mechanism for triggering activity and passing data in a Storyboard application.

Events contain a string name and a data payload. The content of the data payload is described using a format string that allows clients to generically decode the data payload. For example the Lua action performs this decoding to allow access to the event data using string keys.

Events may be received from multiple input sources, but are processed serially by the Storyboard Engine. Clients can quickly and easily define their own custom events to enhance to drive their application and use these in conjunction with the standard events event definitions provided by Storyboard.

A complete list of standard events can be found in the events definition section of this document.

Event Naming Conventions

New events can readily be defined and are not required to contain a data payload, in which case their format string and data payload will be empty values.  When creating new events, it is appropriate to namespace the event definitions so that the names of events do not collide.  For example the Storyboard framework reserves the name prefix of gre. for user interface events and the timer functions all generate events that are prefixed with timer.

The use of events is closely coupled with the declaration and operation of actions.  An action can only be invoked when an event matching the action definition is received.  This results in a common design pattern where an action will perform sophisticated logic in an external script or program and then signal a completion action to run once the script work is complete.

Event >Action (script) >Work >Trigger Event >Action (completion)

Event Format String

The format string on the event is used to assist with the decoding of the event data.  This decoding is used in the action bindings, to allow a minimal amount of additional logic to be placed into the event matching code and also by certain advanced actions, such as Lua scripts, that can symbolically access the event data in the context of the script.

The format string is a description of the data structure passed in the event, it uses entries that are formatted with [numbytes][signed/unsigned][numelements][ ][name]. For the standard C data types the number formatting would look like:

1s0     --> Special null terminated string
1s1|1u1 --> 1 byte integer (int8_t uint8_t)
2s1|2u1 --> 2 byte integer (int16_t uint16_t)
4s1|4u1 --> 4 byte integer (int32_t uint32_t)
8s1|8u1 --> 8 byte integer (int64_t uint64_t)        
                

So if you were transmitting a structure such as:

struct {
int32_t    a;
uint16_t    b;
};
                

You would create a format string containing 4s1 a 2u1 b.  The a and b fields are optional, but they are used to provide the symbolic description of the data to other applications that may want to access it.  This is particularly important if the application incorporates scripting to manage additional complex operations.

Actions

Events trigger actions. Actions do something; manipulate data, interact with the system, log messages, generate more events etc.

Actions can be associated with any model object but they are frequently associated with controls.  When an event is received it is matched first against the action's associated with the currently focused control. After the control's actions the processing passes to the actions associated with the visible layers on the screen followed by the screen and application actions. This cascade of action handling provides an opportunity for the action execution to take place in model context (application, screen, layer, control) that is most appropriate.

Actions always are invoked in response to an event to perform some sort of activity and are always executed in the context of the main execution thread. The trigger event may be a Storyboard standard event or a user event and may come from within the application, from a Lua script action, or be generated from an external program and injected using the Storyboard IO library.

More than one action may be invoked when an event is received.  In this case the order in which the actions are declared within the design is the sequence that they will be invoked when an input event is matched.  This action ordering can be used to ensure an order of operations execution, for example changing the value of a data variable before executing a script that references that data variable.

The following demonstrates how the a gre.press event triggers a gra.datachange action:

For more information on the Storyboard Engine execution pipeline, refer to the Execution Pipeline section of this document.

Actions are extensible components in the Storyboard framework and are usually implemented as plugins to be included in the runtime environment.  Plugins have the ability to hook into the Storyboard Engine managers and can provide a bridge between a Storyboard operation and the embedded system outside of the application.

A complete list of standard actions can be found in the Action Definitions section of this document.

Action Naming Conventions

Actions follow the same naming conventions as data variables, described in the Data Variables section of this document, with the following addition:

The namespace gra. is reserved for Storyboard internal actions.

Data Variables

All user accessible data in Storyboard is maintained in an application database by the Storyboard Engine data manager.   The data manager provides centralized notification when data changes to registered clients within the Storyboard framework.

The data manager stores entries as key/value pairs where the key is a fully qualified model path (string) and the value is a set of bytes with a corresponding format string describing how the data payload should be interpreted.

The Storyboard model is hierarchical, so the construction of a fully qualified model path is straightforward process of joining model element name segments with a . (dot) in between them. The following list demonstrates how the fully qualified model name is formed for a variable, varname, associated with different contexts in the model.

varname

This identifies a variable, varname, as being an application level variable

screen_name.varname

This identifies a variable, varname, as being associated with the screen screen_name

layer_name.varname

This identifies a variable, varname, as being associated with the layer layer_name

screen_name.layer_name.varname

This identifies a variable, varname, as being associated with the layer instance layer_name associated with the screen screen_name

Most variables are not defined as layer instance variables, but rather as layer variables.

layer_name.control_name.varname

This identifies a variable, varname, as being associated with the control control_name that is located on the layer layer_name.

[Note]Note

There is some overlap in the Storyboard namespace that could lead to ambiguous resolution. However control/layer/screen names must be unique which makes the keys in this space non-overlapping. This restriction is enforced by Storyboard Designer.

Using the fully qualified model paths can be cumbersome and impose extra maintenance effort as a project evolves or changes. Storyboard defines several variable shortcuts that will expand their value based on the current context in which they are being resolved.

${app:varname}

Refers to the application variable varname.

${screen:varname}

Refers to the current screen's variable varname

${layer:varname}

Refers to the current layer's variable varname

${control:varname}

Refers to the current control's variable varname

For example, assuming a Storyboard model of:


Application
 + MainScreen
    + FirstLayer
       + AControl
    + SecondLayer
       + AnotherControl
            

where the current focus is associated with the control AControl, then a reference to a variable varname would resolve as follows:

${app:varname}

varname

${screen:varname}

MainScreen.varname

${layer:varname}

FirstLayer.varname

${control:varname}

FirstLayer.AControl.varname

Often these variables are used within render extensions and actions to define access to a dynamic value that will be adjusted at runtime. Alternatively, application variables can be used to provide an application wide setting for a particular value. For example if all text render extensions made their font names be associated with an application variable ${app:heading}, then by changing the value of the heading variable we could change all the fonts in the application.

Storyboard Naming Conventions

Valid data variables and control/layer/screen names must follow the following rules:

  • A valid name matches the following [a-zA-Z]+[a-zA-Z0-9_]*

    • Starts with a character a-z/A-Z not a digit or special character

    • Only '_' is supported as a special character, no spaces in names

  • Screen, Layer, Controls must all be uniquely named

  • User variables cannot collide with a Screen/Layer/Control name

  • The prefix of grd is reserved for Storyboard internal variables

  • The character . (dot) is reserved as a namespace separator

Control and Layer Data Variables

Controls and Layers can be manipulated at runtime by modifying their internal data variables.  These variables are created for each control and layer in the application use the reserved grd_ variable namespace.

These variables are generally accessed using ${model_object:varname}, for example ${control:grd_x} to indicate the x position of the current control

Control variables

The following values can be queried and changed through the normal data management channels.

grd_x (format = 4s1)

The control's x position relative to its layer

grd_y (format = 4s1)

The control's y position relative to its layer

grd_width (format = 4s1)

The control's width

grd_height (format = 4s1)

The control's height

grd_hidden (format = 1u1)

The control's visibility. A value of 0 indicates that the control is visible and 1 that it is hidden

grd_active (format = 1u1)

A value of 0 states that the control is active (can receive and react to events) and 1 for an inactive control (cannot receive or react to events)

grd_opaque (format = 1u1)

Indicates if the control is opaque to events, if opaque (1) then the control will block events from being handled by other controls.  If the value is 0 then the events flow through the control to ones behind it.

Layer variables

The following values can be queried and changed through the normal data management channels.   The position variables are relative to the screen.

grd_x (format = 4s1)

The layer instance's x position relative to the screen

grd_y (format = 4s1)

The layer instance's y position relative to the screen

grd_alpha (format = 1u1)

The layer's transparency value.  The values range from 255 (opaque) to 0 (transparent)

grd_hidden (format = 1u1)

The layer's visibility. A value of 0 states that the layer and all of its controls are visible and a value of 1 hides the layer and all of its controls

Table variables

A table contains all of the control variables and also a set of table specific variables.  These table specific variables can be queried but not dynamically changed.  In order to change these values in a table actions are provided: gra.table.resize, gra.table.scroll.  The variables are as follows.

grd_rows (format = 4s1)

The number of rows in the table

grd_cols (format = 4s1)

The number of columns in the table

grd_visible_rows (format = 4s1)

The number of visible rows in the table

grd_visible_cols (format = 4s1)

The number of visible columns in the table

grd_active_row (format = 4s1)

The row index of the currently active cell

grd_active_col (format = 4s1)

The column index of the currently active cell

grd_row (format = 4s1)

The table’s current top left row

grd_col (format = 4s1)

The table’s current top left column

Maintaining State and Reacting to Changes

In addition to providing a framework for defining the visual display of an application, the Engine framework contains a data manager that is responsible for maintaining the state information that controls the behavior of the application, such as which screens to display and what content should be rendered inside of a control.   The Data Manager contains variables that are user defined (string keys) and typed (string, integer etc) and can be readily modified.

Most often the Data Manager variables are modified by the application itself as it responds to events in the system.   Events are asynchronous triggers, originating internally or externally to the Engine, that are in turn mapped to actions that perform work in the system.  

Actions are where all of the real ‘execution’ is performed within an Engine.  Actions are managed as extensions to the Engine and are flexible.  Some of the built-in actions include changing content in the data manager, manipulating timers, logging messages or triggering additional events both inside and outside of the Storyboard framework.

Execution Pipeline

Below is the execution pipeline that is associated with the internal execution of the Storyboard Engine.

Execution commences with the arrival of an input event to the Storyboard Engine's IO Manager. That event is matched to all of the available actions that are in context by the Action Manager and executed in sequence. Changes in state will result in the a notification to the Screen Manager which will manage the update and refresh of the visual display by invoking the appropriate rendering modules based on the current context.

Trigger Event

While the Engine defines a number of standard user interface types of events, there is no limit to the number of new events that can be created to control custom logic in an application.  There can be multiple event input event sources concurrently generating events, however the events are queued and the delivery of the events, and potential execution of actions as a result, is serialized by the main application thread. This serialization is discussed further in the Execution Environment section of this document.

Action Execution

The application changes its internal state in response to events via actions bound to particular Storyboard model objects.

As an event is processed, it is matched against what events the action handlers available in a particular screen context are expecting.  When there is a match between the received event and the expected event, then the action handler is invoked.  The action handler is invoked with arguments set in the deployment bundle as well as the context of the invocation.  The context includes the current screen, the current and focused controls as well as the event and its payload.

Multiple actions can be bound to a single event.  These actions will be serially executed based on the order in which they were declared in the Design environment. The first declared action is the first action executed.  

Actions can be declared on controls, layers, screens and applications. The event to action matching is performed first for controls based on screen display order (where applicable) and moves sequentially up the stack to process layer, screen and the application actions.  The execution of actions does not stop when one action handler is executed, with the exception that if a control is marked as opaque, then no additional controls will have their action handlers invoked .  If an event is directed (an event which has a position) the only layer which this control is part of will have action handlers invoked, otherwise the actions for all layers will be invoked.  Once control and layer actions have been invoked the following continues to the active screen and application context.  The following diagram illustrates the affects of control flags on action handlers for a directed event (an event which has a position):

The following diagram illustrates the flow for an event which does not have positional information.  These events may be targeted at a specific control or the focused control.

Focus

When an event is received the event is delivered to the currently focused control and this controls layer, and application.  This is true for all events except for directed events such as mouse/touchscreen events which contain positional data.

The focus control for a screen is determined using the focus index value assigned to a control.  A focus index is an integer value which places the control in the screen focus queue.  A focus index must be unique across a screen.  Focus can be navigated via the focus actions which allow for setting focus and moving through the focus queue (next/prev).  The focus actions are described in more detail in the Action Definitions section of this document.

Data Change

The data change is a meta-stage in the execution pipeline.  It is one of many potential outcomes of executing an action handler in response to an event.  However, since most actions cause a change of state, and most state information and variables are maintained in the data manager, this topic deserves special discussion.

The data manager provides clients with the ability to be notified of data content changes through data change listener callback functions.  Since it is possible for multiple action handlers to be invoked as a result of a single event, these callback functions should be restricted to a bare minimum of functionality otherwise they may introduce undesired delay to the graphical rendering operations.

Data change listener notification/processing occurs after all of the actions have been processed for a particular event, allowing data changes resulting from multiple actions to be more efficiently processed.

Display Render

The display render action is also a meta-stage in the execution pipeline.  As actions are executed, if any of these actions cause changes to data or state which is relevant to the rendering of content on the current screen, then the display will be refreshed with the updated content.  If there is no change in data content that would affect the current screen, then there is no display update required.

Execution Environment

A Storyboard application may be multi-threaded, however the execution pipeline is single threaded.  This serialization is provided by the servicing of the event queue as each event is processed through the execution pipeline in sequence.

Internally, within the Engine framework, multiple threads are used to simplify control logic for things such as supporting multiple input sources, or timed event activities.  

The threads that are run as part of the Storyboard framework are generally not signal handling, and mask off all signals.  In certain situations there may be a requirement to handle specific signals, but in those situations the signal handling behavior will be documented as part of the component API.

Animation

Screen Transitions

The most frequently used animation are those that occur during screen transitions.  A screen transition is a way to move from the visible screen to a new screen which may or may not have common layers.  By default screen transitions can be invoked by using one of the following actions:

gra.screen

Transition to a new screen immediately

gra.screen.fade

Fade the new screen into the current screen over time

gra.screen.path

Slide the new screen in and the old screen out over time, this happens from one of the following directions

  • Left

  • Right

  • Top

  • Bottom

gra.screen.scale

Grow the new screen over the current screen

All transition actions which are time based take similar arguments that control the duration of the transition, the rate at which the transitions will occur, the orientation of the transition and the number of frames that should be used.   Using these arguments, the designer can both control the user experience (i.e. duration and effects) as well as the overhead incurred on the system (frequency of frame updates).  

During a screen transition 4 events will be generated to notify the system of the current state.  These events are:

gre.screenshow.pre

This event is generated for the new screen being shown.  The event will be generated before the transition starts. This event gives the user a chance to change data via the gra.datachange action or Lua before ther transition content is updated.

gre.screenhide.pre

This event is generated for the previous screen being hidden.  The event will be generated before the transition starts.

gre.screenshow.post

This event is generated for the new screen being shown.  The event will be generated after the transition has completed

gre.screenhide.post

This event is generated for the previous screen being hidden.  The event will be generated after the transition has completed.

The following illustrates the sequence of events:

The transitions are written such that if graphics hardware layer support is are available, then these layers, assuming they are available for use, will be leveraged to lower the processing overhead for the system during the transition period.  Experience has demonstrated that it is possible to achieve smooth transitions at almost no CPU cost when the hardware capabilities can be properly leveraged.

Timers

Animation can be achieved through the use of timers and data variables.  A control can be moved by changing its position and/or size (ie. ${control:grd_x}) over a period of time.

Animation Action

The Engine internally supports animations through the use of the animation action gra.animate.  This action invokes an animation context which is a list of variables to modify over time at each step of the animation.  As with other animations the rate of change of the variables can also be interpolated over the duration of the animation.

Scripting

The Storyboard action operation behavior based on events and actions provides a limited amount of logic capabilities. When more sophisticated glue logic is needed to control behaviour, a scripting language can be used to interact with the Storyboard environment.

Scripting support is provided through action plugins and there are no limits with respect to which languages may be used. The default scripting language that is provided with the Storyboard framework is Lua (www.lua.org).

Lua was selected for its small footprint, high performance and its ability to be quickly integrated with custom extensions through a well defined C module programming interface.

For more information about the Storyboard Lua integration, refer to the Scripting with Lua section of this document.

External Communication (Storyboard IO)

Communication with external processes in the embedded system can be accomplished in sevaral fashions. One approach that provides a strong API while maintaining a loose coupling for the implementation is to use Storyboard IO.

Storyboard IO, historically known as GREIO, is provided as both a client library and API and as a plugin extension for Storyboard. When the Storyboard IO plugin is loaded a channel is created in order for processes to inject events into the system.  A single event queue is used to serialize the events and therefore any events sent via Storyboard IO will be placed in the queue with standard Storyboard system events.  If the external application wishes to receive events it can create its own Storyboard IO channel which can have events sent through.  Applications can have multiple receive channels and the Engine has a single input channel.  The following diagram illustrates an application which can send events to the Engine and review events on a named channel.

For more details about the client Storyboard IO API, refer to the Storyboard IO API section of this document. For more details about the Storyboard IO action, refer to the Action Definitions section of this document.

Performance Considerations

All actions are executed within the context of an event delivery and as such their execution will have an impact on the overall throughput and responsiveness of the system.  In particular with Lua scripts, it is important to limit the length of time that functions take to perform their work or to separate lengthy operations into separate tasks, threads or processes depending on the operating environment being used.

The screen manager listens for data changes and checks the state of controls to determine when the display needs to be refreshed.  If the data for controls is changing rapidly this may cause thrashing of the display and possible flicker if not using double buffering.   When changing data values, moving controls or generating events which would cause the display to be updated it is advisable to hold the screen manager updates until all changes have been made.  Once modifications are complete the screen manager can be released and the display updated is needed.  The actions are as follows:

  1. gra.screen.hold

  2. gra.screen.release