TORNADO uses Marquise Technologies' STORM API and provides 2 ways of communication with the application:

See below’s chapters for the Storm API documentation.

The interaction between MIST and TORNADO can be for two types of operations:

The creation of a job in TORNADO is a step by step process.

TORNADO allows the management of the jobs: Using Storm API functions like Aborting, Pausing and Deleting are available, together with the list of jobs and their description.

User’s can build their Processing Graphs using the available functionalities in TORNADO.

Some examples of the functions are listed below.

The performances of TORNADO for processing and encoding media highly depend on the capabilities of the hardware chosen. Please make sure to select the server according to your needs.

Depending on the type of format chosen in source and output, TORNADO will use either the CPU or the GPU to decode/encode the file. The default settings for the encoders and decoders can be configured by editing TORNADO’s tornado.cfg configuration file, found in %programdata%/Marquise Technologies/STORM/session and then restarting TORNADO.

Below is an example of a configuration file where the JPEG2000 encoder and decoder are set to use CUDA acceleration and the H264 one is set to CPU:

TORNADO’s decoding and encoding speed will be function of the performances of the CPU and GPU, as well as the disks bandwidth.

If a lot of JPEG2000 processing is foreseen, you may use additional GPU(s) for parallel transcoding.

After the installation of TORNADO, in your web browser enter the following address:

This page allows you to connect your TORNADO and see useful information.

This page displays the list of jobs sent during the current session.

This page lists the errors and warnings for TORNADO managers and workers that are not specific to a particular job.

This information is automatically included when exporting the log of a particular job (see Queue).

TORNADO has its own DCI Certificate enabling the creation of an encrypted DCP. The Certificates page display information about this certificate and also allows to download them.

The exact syntax for the command line interface is explained in the following sections.

When adding a job, the JSON parameters must contain the context definition, otherwise the job will not be added to the queue. Here is an example of how a job context is identified in the JSON parameters:

The values for the context are described in the table below:

For each of the above operatinal contexts, a set of parameters are defined.

To use the command line interface on Microsoft Windows systems, the following steps must be folloed:

Note that if the installation drive is different than the current drive, the current drive (volume) must be changed by typing the drive/volume letter followed by the column character:

Change current drive

Change current directory

The last step is to type the STORM Server command as illustrated above, followed by the operational context and the list of parameters for the context. After this sequence, STORM Server would start in the specified operational context and will continue with the requested processing.

In order to start STORM Server as a REST web service, you must use the following command:

The command above will start the STORM server as a web service listening for HTTP requests on port 8080 by default. This will result in the following:

At this point the server will keep running until the /control/shutdown request is received or the command line process is killed.

If you wish to change the communication port, the following command must be used:

STORM Server can also be installed and started as a system service. In order to do so you must specify the -service flag when installing the server as a service.

The following example uses the Windows sc.exe service manager to install then start STORM Server as a service:

Storm processes media via a list of queued jobs. These jobs can be created directly via the API, or can be spawned automatically by a watchfolder that is actively monitored by a specific node.

Regardless of the manner in which the job has been created, once the job has been added to the queue of jobs, it can be further manipulated like any other regular job.

Processing graphs are complex XML descriptions of various image processing, audio processing and transcoding functions that Storm can apply to the media that needs to be processed.

The processing graph can describe one or more outputs, each of the output being the result of a list of daisy-chained processing nodes.

The following example shows a processing graph document basics (incomplete):

Each node has a unique id, a type and a list of type-specific parameters

Storm can handle a list of watchfolders that are actively monitored.

Every new entry (i.e. new file or new directory) into one of the monitored watch folders is checked by Storm against a list of filters attached to a watchfolder.

If the entry matches one of the filters, then a new job is automatically added to the queue of job. The job is described by a processing graph, associated with the watchfolder.

Compositions are XML timeline descriptions of various audio, video, subtitles and data elements synchronized together. Storm can process complex compositions and apply various processing operators onto them and generate multiple outputs.

Output presets are XML descriptions that define the output format for the input media passed to Storm server. They are templates that can be reused when transcoding.

The following example shows a output preset that describes an MP4 container with a h264 encoded video with a 720p resolution:

For a detailed description of the Output Preset file format see: MT-TN-20 Output Preset File Format Specification

There are various objects that can be manipulated by TORNADO. These objects are entities upon which the process can execute an action, such as jobs, output-presets, watch-folders or workers. Moreover, each object has it’s own dedicated manager. The role of the manager is to handle the object’s lifecycle.

The actions that happen at the manager level and at the object level can generate events. This difference is reflected in the way logs can be accessed via the REST API: The /<object_type>/log endpoint will list all the events that happened at the manager level and the /<object_type><object_UUID>/log will list the events related to a specific object. These events can be used to trace and understand the activity of the system.

For example, in the case of job processing, the object is the job itself while the manager handles the different stages required for executing the type of task represented by the job.

When a job is created, like the case of a transcoding job, a series of steps are executed in order to validate it’s properties and create the delivery engine.

If an error was made in the request structure, like using the wrong execution context, then an error is raised only at the manager level and an event is generated. The event can be consulted by calling the /jobs/log :

If an error was made at the codec quality level when configuring a preset, then an error is raised while validating the codec constraints. This error will generate one or more events at the job object level and can be consulted by calling the /jobs/<job_UUID>/log :

The first event is generated by the codec quality validation failure that, in turn, causes an error in creating the delivery engine which generates another event. These errors will cause the job process to fail. Because job processing includes the validation and engine creation procedures, that event will be logged at the manager level accessible by calling the /jobs/log :

When starting STORM Server in webserver mode, via the websrv operational context, it will automatically accept and handle jobs via an internally managed job queue. This means that the process will handle REST request and jobs on the same node. In this case the job queue is treated sequentially on job at a time.

STORM Server can also be used as a manager node. In this case it has a registered list of worker nodes that handle the received jobs while the manager node is used to distribute jobs and control the worker pool.

A STORM Server worker node can be started as a subprocess of a manager node or as a standalone webserver.

A subprocess worker node can only be created by a manager node and it can only run locally on the same machine as the manager.

A standalone worker node can only be created by an external process and it can run locally or remotely.

In both cases a worker always needs to register with a manager node when started.

In a standard configuration, STORM Server processes media via a list of queued jobs managed on a single, local node. These jobs are handled sequentially, one at a time.

There are cases when multiple nodes are required to handle jobs. In that case some of those nodes will be workers.

Workers can be managed by using the dedicated workers REST API available on the manager node on which they are registered.

Storm maintains a list of compositions that can be obtained at any time. The following example shows you how to obtain the list of compositions registered on a given node:

URL: /compositions METHOD: GET

The following example shows you how to add a new composition to the list managed by a specific node:

The following example shows you how to delete a composition from the list of compositions managed by a specific node:

The Color Management System in TORNADO defines the working color space of the composition. It can be used via the CLI or the REST API for performing colorimetric transformations, like color space conversions when transcoding files.

The CMS works in 2 steps. The first step is determining the source’s properties. TORNADO detects automatically, on a best effort basis, the input’s properties. This step is automatic and can be modified only by using a CutList or an AXF file. The second step is the definition of the target Color Management settings. If not specified then the native settings of the composition, detected during the first step, are used.

In order to override the default target Color Management System a series of parameters must be specified. If not specified the detected source value will be used.

Below is an example of using CMS override parameters, via the CLI or REST APIs, to define the target CMS for a DCP composition used to generate an MPEG4 h264 proxy:

This example will override the detected native CMS settings by defining the target CMS. This will cause a colorimetric transformation using the provided values. In this specific case the proxy will be encoded to the Rec709 color space.

The first step is to specify the CMS system. By default this property has the value NATIVE indicating that the composition’s detected CMS should be used. This parameter should be overriden in order to be able to define a custom workflow. If you select MTCMS, then you must specify the working color space (primaries), the transfer curve (eotf) and the chromatic adaptation (cat). These parameters will define the target CMS.

The transformation to the target CMS is made based on the source parameters. If no metadata is available at the moment of source characterisation then Rec709 is used by default.

Certain values in the file formats specific to TORNADO, like Output Presets or AXFs, use a predefined controlled vocabulary. These values are reflected in the various enumerable objects accessible via the CLI or the REST API. The list of the currently available objects is subject to a continuous revision and extension.

While the structure of an enumerable object might vary slightly, there are element types that are common too all enum objects. One of these elements is the tag type. This type is of particular interest because it is referenced throughout the different technical notes describing the structure of specific file formats like Output Presets or composition AXF files.

The following example shows you how to list the currently available enumerable objects:

STORM Server nodes can be controlled via the dedicated REST API. The API can be used to put in place automated polling mechanisms that check the status, shuts down or restarts nodes.

The following example shows you how to check the status of a node:

Shutting down a node will terminate the process and all managed workers registered with this node.

The following example shows you how to shutdown a node:

Storm maintains a list of jobs that can be obtained at any time. This is the easiest method to get the status of all the running or pending jobs. The following example shows you how to obtain the list of jobs currently running or pending on a given node:

The following example shows you how to retrieve the details for a given job:

The following example shows you how to add a new job to the list of jobs of a specific node:

Running a DELETE HTTP request once will abort the job while running it a second time it will completely remove the job from the internal list.

The following example shows you how to abort or delete a running job for a specific node:

The following example shows you how to obtain a job’s log for a specific object:

The following example shows you how to obtain the list of events that happened at the job manager’s level:

Storm maintains a list of processing graphs that can be obtained at any time.

The following example shows you how to obtain the list of processing graphs currently registered on a given node:

The following example shows you how to add a new processing graph to the list of managed by a specific node:

The following example shows you how to delete a processing graph from the list of processing graphs managed by a specific node:

The following example shows you how to obtain a processing-graph’s log for a specific object:

The following example shows you how to obtain the list of events that happened at the processing-graph manager’s level:

Storm maintains a list of output presets that can be obtained at any time.

The following example shows you how to obtain the list of output presets currently registered on a given node:

The following example shows you how to add a new output preset to the list:

The following example shows you how to delete a output preset from the list of output presets managed by a specific node:

The following example shows you how to obtain an output-preset’s log for a specific object:

The following example shows you how to obtain the list of events that happened at the output-preset manager’s level:

Overlays are XML descriptions that define the format for text and image supposed to be overlayed on top of a transcoding job’s result.

The following example shows an overlay that describes a text element, using a tag element to capture the clip’s name, that is positioned 5% from the left side and 5% from the top side of the image container and is horizontaly left aligned and centered vertically:

For a detailed description of the Overlay file format see: MT-TN-50 Overlay File Format Specification

The following example shows you how to add a new overlay to the list:

The following example shows you how to delete an overlay from the managed list for a specific node:

Storm maintains a list of overlays that can be obtained at any time.

The following example shows you how to obtain the list of overlays currently registered on a given node:

The following example shows you how to obtain an overlay’s log for a specific object:

The following example shows you how to obtain the list of events that happened at the overlay manager’s level:

Storm maintains a list of watchfolders that can be obtained at any time. This is the easiest method to get the status of all the watchfolders that are monitored along with their status.

The following example shows you how to obtain the list of watchfolders currently registered on a given node:

The following example shows you how to add a new watchfolder to the list of watchfolders monitored by a specific node:

The following example shows you how to delete a watchfolder from the list of monitored watchfolders on a specific node:

The following example shows you how to change the status of a watchfoldermonitored for a specific node:

STORM Server maintains a list of managed workers that can be obtained at any time. The following example shows you how to obtain the list of workers being managed by the current node:

The following example describes the API for adding a new worker to the manager’s worker list.

The following example shows you how to delete a specific worker from the manager’s node list.

The following example shows you how to obtain a worker’s log for a specific object:

The following example shows you how to obtain the list of events that happened at the worker manager’s level:

The AssetDescriptorList element is defined by the AssetDescriptorListType and contains an ordered list of AssetDescriptor elements. Each AssetDescriptor element is defined by the AssetDescriptorType.

Below is an example of a AssetDescriptorList:

The ComplianceTestPlanList element is defined by the ComplianceTestListType and contains an ordered list of ComplianceTestPlan elements. Each ComplienceTestPlan element is defined by the ComplianceTestPlanType.

Below is an example of a ComplianceTestPlanList:

The ComplienceTestPlan is defined by the ComplianceTestPlanType and contains an ordered list of TestSequence elements. Each TestSequence element is defined by the TestSequenceType.

Below is an example of a ComplianceTestPlan:

The JobDescriptor element is a complex element containing various sub-elements that describe the nature of the job that produced the XML report.

A UTF-8 text string containing the ISO 639-1 or ISO 639-2 language code

The SubjectDescriptor element is a complex element containing various sub-elements that contain information on the subject that the XML report was produced for.

The values for the various sub-elements are extracted or deduced from the subject’s metadata.

The TestSequenceType contains an ordered list of Test elements. Each Test element is defined by the TestType.

Below is an example of a TestSequence:

When an input file is passed to TORNADO, an internal, virtual composition is built, based on the properties found by decoding the file and the additional parameters specified by the user.

Output Presets are XML files containing elements that define parameters affecting the output of a transcoding job. Their purpose is to specify and reuse the same set of properties for a given master.

The Output Preset document structure is defined with the starting MasterDeliverySpecification root element.

Below is an example of a basic structure of an Output Preset document:

Depending on the desired output not all elements are required to be specified.

The Video element is a complex type using various sub-type elements that describe the video track’s properties. Below is the basic structure of the VideoType:

The Codec element is a complex type using various sub-type elements that describe the video and audio track’s encoding properties.

Below is an example of a definition for the H264 codec using the baseline profile with a bitdepth of 8 and a bitrate of 100 Mbits/s:

The Colorimetry element defines the color-space and the black to white range for the delivery specification. These values are not used for colorimetric transformations but rather to signal the color-space of the delivery specification.

The Audio element is a complex type using various sub-type elements that describe the audio track’s properties. Below is the basic structure of a Audio element:

The SoundfieldList element is a complex type containing a list of Soundfield elements.

The Soundfield element is a complex type using various sub-type elements that describe the properties of a soundfield.

Below is the Soundfield definition where the first 2 channels found in the first audio track in the source composition are mapped to a stereo configuration in the output file.

The ChannelMap element is a complex type containing an ordered list of Channel elements.

Below is an example of a ChannelMap for a 5.1 track configuration:

The Channel element is an unsigned 32-bit integer specifying the source channel to be used. In addition a number of attributes can be specified:

The TimedText element is a complex type using various sub-type elements that describe the properties of timed text tracks.

Below is the basic structure of a TimedText element:

The TrackMap element is a complex type containing an ordered list of Track elements.

The ExtendedProperties element is a complex type using various sub-type elements that describe the additional metadata that should be written into the delivery. Below is the basic structure of a ExtendedProperties element:

The PropertyList element is a complex type containing a list of Property elements.

The Property element is a complex type defining a metadata or additional master delivery property entry as a name/value pair.

The AXF document structure is defined with the starting MTAXF root element, containing an unordered list of Project elements. The most common usage is to store a single Project element per AXF document. Each Project element contains an optional AssetList and an optional CompositionList. Below is the basic structure of an AXF document:

The optional AssetList element is used to reference the assets necessary for the project and the CompositionList element contains an unordered list of compositions. Each composition describes the temporal organization of the assets in a synchronized manner in order to produce a timeline, also know as a play list.

The following section describes the various element and sub-element types used in this specification

TBD

The AssetList element is defined by the AssetListType and contains an ordered list of Asset elements. Each Asset element is defined by the AssetType.

Below is an example of a AssetList:

The AssetType is a complex element type describing the location of the asset within the project’s own library as well as the corresponding location on the file system. In addition to these properties, a globally unique identifier (GUID) is assigned to the asset.