From 02dc8bb63f34b153a4ba352c4f322ffda225abf8 Mon Sep 17 00:00:00 2001
From: Ivan Kondov <ivan.kondov@kit.edu>
Date: Sat, 22 Jul 2017 12:51:41 +0200
Subject: [PATCH] extended spelling, highlighting, markup, style and grammar
---
docs/agenda.rst | 18 +++++------
docs/basics.rst | 75 ++++++++++++++++++++++++----------------------
docs/exercise1.rst | 8 ++---
docs/exercise2.rst | 16 +++++-----
docs/exercise3.rst | 8 ++---
docs/exercise4.rst | 16 +++++-----
docs/exercise5.rst | 38 ++++++++++++-----------
docs/setup.rst | 6 ++--
8 files changed, 97 insertions(+), 88 deletions(-)
diff --git a/docs/agenda.rst b/docs/agenda.rst
index 8527a46..032d5e8 100644
--- a/docs/agenda.rst
+++ b/docs/agenda.rst
@@ -1,31 +1,31 @@
Basic procedures
================
- * Compose fireworks and workflows
+ * Compose Fireworks and workflows
* Validate workflows
* View workflows
- * Add fireworks to LaunchPad: lpad add
- * Execution: rlaunch
- * Monitoring: lpad get_*, lpad webgui
+ * Add fireworks to LaunchPad: ``lpad add``
+ * Execution: ``rlaunch``
+ * Monitoring: ``lpad get_*``, ``lpad webgui``
Exercise 1: Managing control flow
=================================
* Dependencies and concurrency
- * Use standard firetasks: ScriptTask
+ * Use standard Firetasks: ``ScriptTask``
* Example: F1 pitstop
Exercise 2: Managing data flow
==============================
* Data flow dependencies
- * Use custom firetasks: PythonFunctionTask
+ * Use custom Firetasks: ``PythonFunctionTask``
* Example: Recruiting
Exercise 3: Manage data in files and command line input
=======================================================
- * Use custom firetasks: CommandLineTask
+ * Use custom Firetasks: ``CommandLineTask``
* Example: Image reconstruction
Exercise 4: Extending a workflow
@@ -34,7 +34,7 @@ Exercise 4: Extending a workflow
* Extension of example from Exercise 3
* Example: Image swirl
-Execise 5: Writing a Firetask
-=============================
+Exercise 5: Writing a Firetask
+==============================
* Extension of example from Exercise 2
diff --git a/docs/basics.rst b/docs/basics.rst
index 83198a6..5de87b2 100644
--- a/docs/basics.rst
+++ b/docs/basics.rst
@@ -10,11 +10,10 @@ Compose fireworks and workflows
Formats and editors
~~~~~~~~~~~~~~~~~~~
-Fireworks and Workflows can be defined in three different general-purpose
-languages: Python, JSON and YAML. There is no domain-specific language for
+Fireworks and workflows can be defined in three different general-purpose
+languages: Python, JSON and YAML. There is no domain-specific language in
FireWorks and thus no specialized editor. This is why, a normal text editor is
-sufficient and an IDE supporting editing and validation of these three languages
-is "nice to have".
+sufficient.
All workflow definitions in all exercises will be based on JSON and YAML.
Exercise 5 will introduce to writing a custom Firetask for which
@@ -31,17 +30,17 @@ JSON versions of the demos and solutions.
Workflow structure
~~~~~~~~~~~~~~~~~~
-The building blocks of a workflow are the fireworks. The firework is the minimum
-possible piece of the worklfow executed by the rocket launcher (see execution_).
-The major part of the workflow description is the list of fireworks, ``fws``.
-A firework has an ID ``fw_id``, a name ``name``, and a specification ``spec``
-containing a list of firetasks and firework specific data. In addition, a
-dictionary named ``links`` with dependencies between the fireworks has to be
+The building blocks of a workflow are Fireworks. A Firework is the minimum
+possible piece of the workflow executed by the rocket launcher (see below).
+The major part of the workflow description is the list of Fireworks, ``fws``.
+A Firework has an ID ``fw_id``, a name ``name``, and a specification ``spec``
+containing a list of Firetasks and Firework specific data. In addition, a
+dictionary named ``links`` with dependencies between the Fireworks has to be
specified. Further attributes are ``metadata`` and ``name``.
-Each firetask includes the firetask name ``_fw_name`` and definitions of its
-parameters. Every firetask is *atomic* i.e. executed at once without further
-subdivisions. The firetasks of one firework are executed strictly one after
+Every Firetask includes the Firetask name ``_fw_name`` and definitions of its
+parameters. Firetask is *atomic* i.e. executed at once without further
+subdivisions. The Firetasks of one Firework are executed strictly one after
another in the order of their specification and share the same job working
directory and the files in it.
@@ -79,17 +78,20 @@ command::
yaml2json < workflow.yaml > workflow.json
-Add fireworks to LaunchPad
+Add Fireworks to LaunchPad
--------------------------
The LaunchPad is a database where the workflows are stored during their full
-life cycle.
+life cycle. It is hosted on a resource named FireServer.
+
+**NOTE:** For simplicity, in this tutorial the FireServer is the same host on
+which you are logged on.
When used productively the LaunchPad contains many workflows in different states.
To distinguish between different workflows, the query commands can specify e.g.
-the firework ID from the relevant workflow on the LaunchPad or perform a
+the Firework ID from the relevant workflow on the LaunchPad or perform a
mongo-like queries. To avoid the need to apply filters to the queries, we will
-clean up the LaunchPad from previous fireworks at the beginning of each exercise
+clean up the LaunchPad from previous Fireworks at the beginning of each exercise
in this tutorial with this command::
lpad reset
@@ -140,27 +142,30 @@ Execute workflows
-----------------
The workflow engine of FireWorks is called FireWorker. Multiple FireWorkers can be
-runninng on different resources where idividual fireworks can be executed by the
+running on different resources where individual Fireworks can be executed by the
rocket launcher ``rlaunch`` which has three modes of operation: *singleshot*,
*rapidfire* and *multi*.
-To only execute one firework from the LaunchPad which is in *READY* state and
-exit the following command is used::
+**NOTE:** For simplicity, in this tutorial the FireWorker is the same host as
+the FireServer.
+
+To only execute one Firework from the LaunchPad which is in *READY* state the
+following command is used::
rlaunch singleshot
-To run all fireworks in *READY* state in a sequence::
+To run all Fireworks in *READY* state in a sequence::
rlaunch rapidfire
-**NOTE:** Every firework changes its state to *READY* after all its parent
-fireworks are completed (state *COMPLETED*) and the states of linked child
-fireworks are updated as soon as a firework if completed. This means that any
-workflow will be run until there are no more fireworks in *READY* state.
+**NOTE:** Every Firework changes its state to *READY* after all its parent
+Fireworks are completed (state *COMPLETED*) and the states of linked child
+Fireworks are updated as soon as a Firework if completed. This means that any
+workflow will be run until there are no more Fireworks in *READY* state.
-**NOTE:** In singleshot mode ``rlaunch`` runs the firework in the directory
-where it is started. In rapidfire mode ``rlaunch`` creates separate per-firework
-sub-directories at run time in which the fireworks are executed.
+**NOTE:** In *singleshot* mode ``rlaunch`` runs the Firework in the directory
+where it is started. In rapidfire mode ``rlaunch`` creates separate
+sub-directories for each Firework at run time.
To suppress verbose information on the screen the *-s* flag can be added::
@@ -174,7 +179,7 @@ To query workflows available on the LaunchPad use the command ``lpad get_wflows`
lpad get_wflows [-d <more|all>]
-To query individual fireworks use the command::
+To query individual Fireworks use the command::
lpad get_fws [-i <firework ID> [-d <more|all>]]
@@ -189,11 +194,11 @@ X server is running on your machine and it is configured and tunneled properly
in your terminal session.
-Remove a worklfow from LaunchPad
+Remove a workflow from LaunchPad
--------------------------------
A selection of workflows can be deleted from LaunchPad using the ``lpad delete_wflows``
-command. For example to delete worklfows including fireworks with given IDs::
+command. For example to delete workflows including Fireworks with given IDs::
lpad delete_wflows -i <firework IDs>
@@ -201,13 +206,13 @@ command. For example to delete worklfows including fireworks with given IDs::
Failure and restart
-------------------
-If execution of a firework fails for some reason, its state changes to ``FIZZLED``.
-The failure reason can be found in the launch section of the firework using the
+If execution of a Firework fails for some reason, its state changes to ``FIZZLED``.
+The failure reason can be found in the launch section of the Firework using the
command::
lpad get_fws -i <firework ID> -d all
-If the reason is external and the error is solved then the firework can be rerun
+If the reason is external and the error is solved then the Firework can be rerun
with::
- lpad rerun_fws -i <firework ID>
\ No newline at end of file
+ lpad rerun_fws -i <firework ID>
diff --git a/docs/exercise1.rst b/docs/exercise1.rst
index d151e5d..94f1585 100644
--- a/docs/exercise1.rst
+++ b/docs/exercise1.rst
@@ -2,7 +2,7 @@ Exercise 1: Managing control flow
=================================
With this exercise we will learn to describe the dependencies in control flow
-between fireworks, exploit possible concurrencies and use the standard firetask
+between Fireworks, exploit possible concurrencies and use the standard Firetask
``ScriptTask``.
Problem 1.1
@@ -21,10 +21,10 @@ and then find and correct the errors.
Add the workflow to the LaunchPad and query its state.
-Execute the workflow with ``rlaunch singleshot``, i.e. run firework at a time.
+Execute the workflow with ``rlaunch singleshot``, i.e. run one Firework at a time.
-After running each single firework, monitor the output and the states of the
-fireworks until the workflow is completed.
+After running each single Firework, monitor the output and the states of the
+Fireworks until the workflow is completed.
Problem 1.2
diff --git a/docs/exercise2.rst b/docs/exercise2.rst
index 0535a1f..cfb8385 100644
--- a/docs/exercise2.rst
+++ b/docs/exercise2.rst
@@ -1,8 +1,8 @@
Exercise 2: Managing data flow
==============================
-The purpose of this exercise is to learn how to pass data between fireworks and
-describe data dependencies using the custom firetask ``PythonFunctionTask``::
+The purpose of this exercise is to learn how to pass data between Fireworks and
+describe data dependencies using the custom Firetask ``PythonFunctionTask``::
- _fw_name: PythonFunctionTask
function: any_module.any_function
@@ -12,10 +12,10 @@ describe data dependencies using the custom firetask ``PythonFunctionTask``::
outputs:
- data to forward
-The keys specified in the ``inputs`` list must be available in the firework
+The keys specified in the ``inputs`` list must be available in the Firework
``spec`` at task run time. These are passed as positional arguments to the
-function. The returned outputs will be stored in the specs of the current firework
-and of the child fireworks under the keys specified in the ``outputs`` list.
+function. The returned outputs will be stored in the specs of the current Firework
+and of the child Fireworks under the keys specified in the ``outputs`` list.
Problem 2.1
@@ -24,8 +24,8 @@ Problem 2.1
Change to folder **exercises/work/2_data_flow**. Copy the provided template
**exercises/inputs/2_data_flow/template.[json|yaml]** and complete it so that
the script **exercises/problems/recruiting-script.py** is implemented as a
-workflow. Check the workflow, add it to LaunchPad and run it in singleshot mode
-watching the changes in the fireworks with the workflow run.
+workflow. Check the workflow, add it to LaunchPad and run it in *singleshot* mode
+watching the changes in the Fireworks with the workflow run.
Problem 2.2
@@ -33,6 +33,6 @@ Problem 2.2
Copy the different solutions **recruiting-[012].[json|yaml]** from the folder
**exercises/problems/2_data_flow**, and detect and correct the errors and run
-the workflow in rapidfire mode. Compare the corrected versions to each other and
+the workflow in *rapidfire* mode. Compare the corrected versions to each other and
to your solution for Problem 2.1. Compare the results of two instances of the
same workflow. Why do they differ?
diff --git a/docs/exercise3.rst b/docs/exercise3.rst
index 5cfaee9..62c793a 100644
--- a/docs/exercise3.rst
+++ b/docs/exercise3.rst
@@ -1,10 +1,10 @@
Exercise 3: Manage data in files and command line input
=======================================================
-The purpose of this exercise is to learn how to pass data between fireworks as
+The purpose of this exercise is to learn how to pass data between Fireworks as
files and process these data using the custom ``CommandLineTask``. The built-in
``ScriptTask`` used in **Exercise 1** allows to run a script but provides no
-methods to move data between fireworks and no handling of command line options, flags
+methods to move data between Fireworks and no handling of command line options, flags
input and output as workflow data.
Given is a set of reusable operations implemented using the ``convert`` and
@@ -18,7 +18,7 @@ command line flags. These are:
- Swirl
- Animate
-The fireworks corresponding to these operations can be found in
+The Fireworks corresponding to these operations can be found in
**exercises/demos/3_files_and_commands**. The provided solutions in
**exercises/solutions/3_files_and_commands** are suitable for the input for
letter "A" in **exercises/inputs/3_files_and_commands/A**.
@@ -34,7 +34,7 @@ selected letter.
Select a set of input images (**piece-1.png** and **piece-2.png**) for a letter
from the folder **exercises/inputs/3_files_and_commands**.
-Then use the provided fireworks and compose a workflow to reconstruct the
+Then use the provided Fireworks and compose a workflow to reconstruct the
selected letter by adjusting the image processing parameters, inputs and outputs.
Verify, add and run the workflow and check the resulting image.
diff --git a/docs/exercise4.rst b/docs/exercise4.rst
index 881b8e4..d3072b5 100644
--- a/docs/exercise4.rst
+++ b/docs/exercise4.rst
@@ -3,7 +3,7 @@ Exercise 4: Extending a workflow
Workflows in FireWorks can be extended at any time of their life cycle,
particularly in states *READY*, *RUNNING* and *COMPLETED*. If the appended
-firework or workflow is appended as a detour then the child fireworks may not be
+Firework or workflow is appended as a detour then the child Fireworks may not be
be *RUNNING* or *COMPLETED*.
@@ -17,10 +17,10 @@ latter workflow. For this, first copy the workflow **image_swirl.json** from
**exercises/solutions/3_files_and_commands** to a file
**image_swirl_montaged.json**. Then do the following changes:
-- Change all firework IDs to become negative integers. This is required by the
+- Change all Firework IDs to become negative integers. This is required by the
``append_wflow`` method.
-- Add the following firework to the list of fireworks (for JSON adapt the code
+- Add the following Firework to the list of Fireworks (for JSON adapt the code
correspondingly)::
- fw_id: -6
@@ -36,18 +36,18 @@ latter workflow. For this, first copy the workflow **image_swirl.json** from
'-6': [-1, -2, -3, -4, -5]
to the ``links`` dictionary. Why is this step necessary?
-- Modify it so that the input ``original image`` in all fireworks is passed from
- a parent firework::
+- Modify it so that the input ``original image`` in all Fireworks is passed from
+ a parent Firework::
original image: {source: montaged image}
- Save the thus prepared sub-workflow file as **image_swirl_montaged.[yaml|json]**.
-Now identify the firework ID of the firework providing the final image as output::
+Now identify the Firework ID of the Firework providing the final image as output::
lpad get_fws -n "Put the four pieces together"
-Alternatively, the workflow can be found by name and the last firework can be
+Alternatively, the workflow can be found by name and the last Firework can be
found in the more detailed printout::
lpad get_wflows -n "Image reconstruction" -d more
@@ -61,4 +61,4 @@ is now marked from *COMPLETED* to *RUNNING*::
lpad get_wflows -s "RUNNING"
-Run the extended workflow.
\ No newline at end of file
+Run the extended workflow.
diff --git a/docs/exercise5.rst b/docs/exercise5.rst
index 091c3b6..fff1418 100644
--- a/docs/exercise5.rst
+++ b/docs/exercise5.rst
@@ -3,20 +3,22 @@ Exercise 5: Writing a Firetask
This exercise bases on an extended version of the example from **Exercise 2**.
It is implemented as a python script
-**exercises/problems/5_author_firetask/recruiting-script.py**. It has two new aspects:
+**exercises/problems/5_author_firetask/recruiting-script.py**. It has two new
+aspects:
1. All parameters are loaded at the beginning from a JSON document
**parameters.json**.
- 2. The steps candidates_apply() and screen_candidates() are repeated as long
- as the size of the canditate list is smaller than ``number_to_invite``.
+ 2. The steps ``candidates_apply()`` and ``screen_candidates()`` are
+ repeated as long as the size of the candidate list is smaller than
+ ``number_to_invite``.
Anatomy of Firetasks
--------------------
-All firetasks are classes derived from the ``FiretaskBase`` class. Many built-in
-Firetasks are available in the upstream FireWorks, such as the ``ScriptTask``,
-``PyTask``, ``FileTransferTask``. Additional firetasks can be written for both
+All Firetasks are classes derived from the ``FiretaskBase`` class. Many built-in
+Firetasks are available in the upstream Fireworks, such as the ``ScriptTask``,
+``PyTask``, ``FileTransferTask``. Additional Firetasks can be written for both
generic and specific purposes.
The skeleton of a Firetask looks like this::
@@ -42,7 +44,7 @@ The skeleton of a Firetask looks like this::
workflow.
"""
- print(self[par1], self[par2]) # print the values of required params
+ print(self[par1], self[par2]) # print the values of required parameters
print(self.fw_spec['name']) # print the name of the Firework
if self.get('optional par'):
@@ -57,7 +59,7 @@ The skeleton of a Firetask looks like this::
return FWAction(*actions)
-Here a firework using this firetask::
+Here a Firework using this Firetask::
- fw_id: 1
name: Sample firework
@@ -78,10 +80,10 @@ $PYTHONPATH with::
export PYTHONPATH=`pwd`:$PYTHONPATH
-Problem 5.1: Data Loader Task
------------------------------
+Problem 5.1: Data Loader Firetask
+---------------------------------
-Write a DataLoaderTask to load the necessary parameters from the JSON document.
+Write a ``DataLoaderTask`` to load the necessary parameters from the JSON document.
If necessary the input data structure can be split into more than one file.
The resulting workflow, which is given in **dataloader.json**, must be successfully
running with no further modifications.
@@ -91,16 +93,16 @@ JSON documents as list or dictionaries and then return a ``FWAction`` object
with ``update_spec`` and the structure (see above example).
-Problem 5.2: Conditional Repeater Task
---------------------------------------
+Problem 5.2: Conditional Repeater Firetask
+------------------------------------------
-Write a ``RepeatIfLengthLesser`` firetask that can implement the while loop in the
-script. The firetask should integrate into the workflow available in
+Write a ``RepeatIfLengthLesser`` Firetask that implements the while loop in the
+script. The Firetask should integrate into the workflow available in
**dataloader+repeater.json** without further adaptations.
**Hint:** You can use the ``load_object`` function from
-``fireworks.utilities.fw_serializers`` to construct a firework object and
-the ``detours`` keyword argument of ``FWAction`` to insert the firework
+``fireworks.utilities.fw_serializers`` to construct a ``Firework`` object and
+the ``detours`` keyword argument of ``FWAction`` to insert the Firework
dynamically::
firework = Firework(
@@ -108,4 +110,4 @@ dynamically::
spec=fw_spec,
name='repeat '+self['measure']
)
- return FWAction(detours=firework)
\ No newline at end of file
+ return FWAction(detours=firework)
diff --git a/docs/setup.rst b/docs/setup.rst
index 3553b83..1fb25a7 100644
--- a/docs/setup.rst
+++ b/docs/setup.rst
@@ -4,7 +4,8 @@ Setup
Prerequisites
-------------
-* Python
+Python
+~~~~~~
For this tutorial it is recommended to use the anaconda python distribution.
Download anaconda (Python 3.6 version) from https://www.continuum.io/downloads
@@ -27,7 +28,8 @@ occurs, then the libgcc package has to be installed/upgraded::
conda install libgcc
-* MongoDB
+MongoDB
+~~~~~~~
If your system is Ubuntu and you have administrator permissions you can install
MongoDB system-wide with the following command::
--
GitLab