basics.rst

Basic procedures

Compose fireworks and workflows

Fireworks and Workflows can be defined in three different general-purpose languages: 1) Python 2) JSON and 3) YAML. There is no domain-specific language for FireWorks and thus no specialized editor. Thus, a normal text editor is sufficient and an IDE supporting editing and validation of these three languages is "nice to have".

All workflow definitions in all exercises will be based on JSON and YAML. Exercise 5 will introduce to writing a custom Firetask for which Python will be used. Again, JSON and YAML will be used to define the workflows using the custom Firetask. For each exercise there are one or more initial examples in the exercises/demos folder. We recommend trying these examples before starting solving the problems.

A firework consists of <describe the constituting parts: Firetask, Fireworks and relevant attributes>

Add fireworks to LaunchPad

The LaunchPad is a database where the workflows are stored during their full life cycle.

When used productively the LaunchPad contains many workflows in different states. To distinguish between different workflows, the query commands can specify e.g. the firework ID from the relevant workflow on the LaunchPad or perform a mongo-like queries. To avoid the need apply filters to the queries, at the beginning of each exercise in this tutorial we will clean up the LaunchPad from previous fireworks with this command:

lpad reset

To add a workflow to the LaunchPad:

lpad add workflow.yaml

Alternatively in JSON format:

lpad add workflow.json

Validate workflows

Formal verification is done with adding a workflow to the LaunchPad. However, missing links or data dependencies, and circular dependencies are not detected at this stage and the errors appear at run time. To also check for such errors the -c or --check flags can be used when adding the workflow to the LaunchPad:

lpad add -c workflow.json

If a workflow has been added without check it can be check later with:

lpad check_wflow -i <firework ID>

Note: The correctness check is recommended for all exercises in the tutorial.

View workflows

Already added workflows can be converted into DOT format and viewed graphically as PDF:

lpad check_wflow -i <firework ID> [--view_control_flow] [--view_data_flow] [-f <DOT_FILE>]

Execute workflows

The workflow engine of FireWorks is called FireWorker. Multiple FireWorkers can be runninng on different resources where idividual fireworks can be executed by the rocket launcher rlaunch which has three modes of operation: singleshot, rapidfire and multi.

To only execute one firework from the LaunchPad which is in READY state and exit the following command is used:

rlaunch singleshot

To run all fireworks in READY state in a sequence:

rlaunch rapidfire

Note: Every firework changes its state to READY after all its dependency fireworks are completed (state COMPLETED) and the states of linked fireworks are updated as soon as a firework if completed. This means that any workflow will be continuous executed until there are no fireworks in READY state.

To suppress verbose information on the screen the -s flag can be added:

rlaunch -s rapidfire

Monitor workflows

To query workflows available on the LaunchPad use the command get_wflows:

lpad get_wflows [-d <more|all>]

To query individual fireworks use the command:

lpad get_fws [-i <firework ID> [-d <more|all>]]

Note: The query from the command line is recommended for thi tutorial.

Alternatively the web GUI can be used:

lpad webgui

Note: Make sure that a web browser is configured in your terminal session and an X server is running on your machine and it is configured and tunneled properly in your terminal session.