From 168e8a21e0987dfa5bc1b9d2a0737fdb4f69e437 Mon Sep 17 00:00:00 2001
From: Ivan Kondov <ivan.kondov@kit.edu>
Date: Thu, 14 Nov 2019 11:27:02 +0100
Subject: [PATCH] added demo on control flow, adapted LICENSE and README,
extended docs
---
LICENSE | 2 +-
README.rst | 17 +++++++--------
demos/1_control_flow/hamlet.json | 35 ++++++++++++++++++++++++++++++
demos/1_control_flow/hamlet.py | 19 ++++++++++++++++
demos/1_control_flow/hamlet.yaml | 15 +++++++++++++
docs/basics.rst | 37 +++++++++++++++++++++-----------
docs/setup.rst | 12 ++++++-----
7 files changed, 109 insertions(+), 28 deletions(-)
create mode 100644 demos/1_control_flow/hamlet.json
create mode 100644 demos/1_control_flow/hamlet.py
create mode 100644 demos/1_control_flow/hamlet.yaml
diff --git a/LICENSE b/LICENSE
index ab3d41f..16a768b 100644
--- a/LICENSE
+++ b/LICENSE
@@ -1,6 +1,6 @@
BSD 3-Clause License
-Copyright (c) 2017, Karlsruhe Institute of Technology
+Copyright (c) 2017-2019, Karlsruhe Institute of Technology
All rights reserved.
Redistribution and use in source and binary forms, with or without
diff --git a/README.rst b/README.rst
index d934adc..c7d4314 100644
--- a/README.rst
+++ b/README.rst
@@ -5,13 +5,12 @@ provenance, provide methods for validation and error tracking, and exploit
application concurrency using distributed computing resources. The goal of this
tutorial is to learn composing and running workflow applications using the
FireWorks workflow environment (https://materialsproject.github.io/fireworks/).
-In the first part, after an introduction to the concept of workflows, to
-state-of-the-art workflow systems and to FireWorks, the participants will learn
-to construct workflows using a library of existing Firetasks. The composed
-workflows will be verified, visualized and then executed and monitored. The
-exercises include managing data and control flow dynamically using
-FWAction. The last part focuses on writing custom Firetasks to match more
-specific application requirements.
+After an introduction to the FireWorks, the participants will learn to construct
+workflows using a library of existing Firetasks. The composed workflows will
+be verified, visualized and then executed and monitored. The exercises include
+managing data and control flow dynamically using FWAction. The goal of Exercise
+5 is to learn writing custom Firetasks to match more specific application
+requirements.
-Basic knowledge of using the bash shell is required. For the last part, basic
-knowledge of Python is required.
\ No newline at end of file
+Basic knowledge of using the bash shell is required. For Exercise 5 basic
+knowledge of Python is required.
diff --git a/demos/1_control_flow/hamlet.json b/demos/1_control_flow/hamlet.json
new file mode 100644
index 0000000..3d38019
--- /dev/null
+++ b/demos/1_control_flow/hamlet.json
@@ -0,0 +1,35 @@
+{
+ "fws": [
+ {
+ "fw_id": 1,
+ "name": "First act",
+ "spec": {
+ "_tasks": [
+ {
+ "_fw_name": "ScriptTask",
+ "script": "echo 'To be, or not to be,'"
+ }
+ ]
+ }
+ },
+ {
+ "fw_id": 2,
+ "name": "Second act",
+ "spec": {
+ "_tasks": [
+ {
+ "_fw_name": "ScriptTask",
+ "script": "echo 'that is the question:'"
+ }
+ ]
+ }
+ }
+ ],
+ "links": {
+ "1": [
+ 2
+ ]
+ },
+ "metadata": {},
+ "name": "Hamlet workflow"
+}
diff --git a/demos/1_control_flow/hamlet.py b/demos/1_control_flow/hamlet.py
new file mode 100644
index 0000000..a1e3fd3
--- /dev/null
+++ b/demos/1_control_flow/hamlet.py
@@ -0,0 +1,19 @@
+from fireworks import Firework
+from fireworks import Workflow
+from fireworks import ScriptTask
+from fireworks import LaunchPad
+from fireworks.core.rocket_launcher import rapidfire
+
+# construct the workflow
+fw1 = Firework(ScriptTask.from_str('echo "To be, or not to be,"'),
+ name='First act')
+fw2 = Firework(ScriptTask.from_str('echo "that is the question:"'),
+ name='Second act')
+workflow = Workflow([fw1, fw2], links_dict={fw1: fw2}, name='Hamlet workflow')
+
+# add the workflow to launchpad
+launchpad = LaunchPad.from_file('launchpad.yaml')
+launchpad.add_wf(workflow)
+
+# run the workflow
+rapidfire(launchpad)
diff --git a/demos/1_control_flow/hamlet.yaml b/demos/1_control_flow/hamlet.yaml
new file mode 100644
index 0000000..3dda22e
--- /dev/null
+++ b/demos/1_control_flow/hamlet.yaml
@@ -0,0 +1,15 @@
+fws:
+- fw_id: 1
+ name: First act
+ spec:
+ _tasks:
+ - {_fw_name: ScriptTask, script: 'echo ''To be, or not to be,'''}
+- fw_id: 2
+ name: Second act
+ spec:
+ _tasks:
+ - {_fw_name: ScriptTask, script: 'echo ''that is the question:'''}
+links:
+ '1': [2]
+metadata: {}
+name: Hamlet workflow
diff --git a/docs/basics.rst b/docs/basics.rst
index e2b01a9..e673f89 100644
--- a/docs/basics.rst
+++ b/docs/basics.rst
@@ -22,10 +22,10 @@ using custom Firetasks. For each exercise there are one or more initial
examples in the **demos** folder. Trying these examples is recommend
before starting solving the problems.
-**NOTE:** Most of the examples here will be presented in YAML (more readable and
-concise). If you feel more comfortable with editing JSON you can use the converters
-``yaml2json`` and ``json2yaml`` provided in the **bin** folder or/and the provided
-JSON versions of the demos and solutions.
+**NOTE:** Most of the examples here will be presented in YAML (more readable
+and concise). If you feel more comfortable with editing JSON you can use the
+converters ``yaml2json`` and ``json2yaml`` provided in the **bin** folder
+or/and the provided JSON versions of the demos and solutions.
Workflow structure
~~~~~~~~~~~~~~~~~~
@@ -159,7 +159,7 @@ To run all Fireworks in *READY* state in a sequence::
**NOTE:** Every Firework changes its state to *READY* after all its parent
Fireworks are completed (state *COMPLETED*) and the states of linked child
-Fireworks are updated as soon as a Firework if completed. This means that any
+Fireworks are updated as soon as a Firework is completed. This means that any
workflow will be run until there are no more Fireworks in *READY* state.
**NOTE:** In *singleshot* mode ``rlaunch`` runs the Firework in the directory
@@ -171,8 +171,8 @@ To suppress verbose information on the screen the *-s* flag can be added::
rlaunch -s rapidfire
-Query workflows
----------------
+Query workflows and fireworks
+-----------------------------
To query workflows available on the LaunchPad use the command
``lpad get_wflows``::
@@ -181,19 +181,19 @@ To query workflows available on the LaunchPad use the command
The ID of any firework included in the workflow can be used to query a
specific workflow. Alternatively, workflows can be filtered using a pymongo
-query after the **-q** flag.
+query after the *-q* flag.
To query individual Fireworks use the command::
lpad get_fws [-i <firework ID>] [-d <more|all>]
-Adding the flag **-o yaml** after **lpad** will produce the output in YAML
+Adding the flag **-o yaml** after ``lpad`` will produce the output in YAML
instead of JSON. To obtain a more detailed help on a specifc lpad command you
can the online help::
lpad <lpad command> --help
-The following will provide a full list of commands and **lpad** options::
+The following will provide a full list of commands and ``lpad`` options::
lpad --help
@@ -211,13 +211,24 @@ Fireworks with given IDs::
Failure and restart
-------------------
-If execution of a Firework fails for some reason, its state changes to ``FIZZLED``.
+If an execution of a Firework fails for some reason, its state changes to ``FIZZLED``.
The failure reason can be found in the launch section of the Firework using the
command::
lpad get_fws -i <firework ID> -d all
-If the reason is external and the error is solved then the Firework can be rerun
-with::
+If the reason is an external error that is solved then the Firework can be
+rerun with::
lpad rerun_fws -i <firework ID>
+
+If the reason is an error in the ``spec`` then the command ``lpad update_fws``
+can be used to correct the error::
+
+ lpad update_fws -i <firework ID> -u '{<your update>}'
+
+Then the firework can be rerun.
+
+**NOTE:** Re-running a Firework does not reset any dynamic actions taken by
+that Firework, such as creating new Workflow steps or modifying the **spec**
+of its children.
diff --git a/docs/setup.rst b/docs/setup.rst
index 849ed94..89659b7 100644
--- a/docs/setup.rst
+++ b/docs/setup.rst
@@ -54,7 +54,7 @@ occurs, then the libgcc package has to be installed/upgraded::
Setting up a FireServer
~~~~~~~~~~~~~~~~~~~~~~~
-The tutorial requires a FireServer which is accessible from your computer.
+The tutorial requires a FireServer which is accessible from your computer and from your computing resource (e.g. an HPC cluster).
Remote FireServer
'''''''''''''''''
@@ -78,21 +78,23 @@ necessary). The file must have the following contents::
The root CA certficate can be downloaded from this URL
http://www.gridka.de/ca/dd4b34ea.pem and installed locally.
-The hostname, username and password you will get from the instructors. In the
+You will get the hostname, username and password from the instructors. In the
folder **$HOME/.fireworks** the fireworks configuration file
**FW_config.yaml** must be created with the following content::
echo LAUNCHPAD_LOC: $HOME/.fireworks/launchpad.yaml >> .fireworks/FW_config.yaml
-This configuration has to made on all resources that will use the FireServer, i.e.
-your local computer, login nodes and compute nodes of the clusters etc.
+This configuration has to made available on all resources that will use the
+FireServer, i.e. your local computer, login nodes and compute nodes of the
+clusters etc. If these resources share the **$HOME** file system then the
+configuration has to be done only once.
Local FireServer
''''''''''''''''
If you have Linux and administrator permissions on your computer you can install
-MongoDB system-wide with the following command::
+MongoDB system-wide, e.g. on Ubuntu with the following command::
sudo apt-get install mongodb
--
GitLab