Compare revisions

Ivan Kondov · Ivan Kondov · Ivan Kondov · Ivan Kondov · Ivan Kondov · 4f25e184
--- a/docs/tools.md
+++ b/docs/tools.md
@@ -28,7 +28,7 @@ By using the `-h, --help` flag a summarized help can be obtained:
 ```
 texts script --help
-usage: texts script [-h] -f MODEL_FILE [-g GRAMMAR_PATH] [-m {instant,deferred,workflow}] [-t] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-d] [-q QADAPTER_FILE] [-w] [--no-unique-launchdir] [--enable-logging]
+usage: texts script [-h] -f MODEL_FILE [-g GRAMMAR_PATH] [-m {instant,deferred,workflow}] [-l LAUNCHPAD_FILE] [-u UUID] [-r] [-d] [-q QADAPTER_FILE] [-w] [--no-unique-launchdir] [--enable-logging]
                    [--logging-level {NOTSET,DEBUG,INFO,WARNING,ERROR,CRITICAL}] [--no-duplicate-detection]
 Run a script
@@ -41,7 +41,6 @@ options:
                        path to grammar root file
  -m {instant,deferred,workflow}, --mode {instant,deferred,workflow}
                        execution mode
-  -t, --textx-debug     enable textx debug mode
  -l LAUNCHPAD_FILE, --launchpad-file LAUNCHPAD_FILE
                        path to launchpad file (workflow mode only)
  -u UUID, --uuid UUID  UUID of a model to extend/run (workflow mode only)
@@ -68,7 +67,7 @@ By default, in workflow mode every statement will be evaluated in a unique launc
 The interactive session, started with `texts session`, is a tool managing a session for processing a model in *workflow mode*. Some of the startup flags are the same as for `texts script`.
-By default no evaluation is performed. The evaluation with the `-r, --autorun` without the `-a, --async-run` flag is effective only for workflow nodes of [_interactive_ category](resources.md#policies-for-interactive-and-batch-execution). The flag `-a, --async-run` activates background evaluation using a workflow engine with a launcher thread. The evaluation includes workflow nodes of both [_interactive_ and _batch_ categories](resources.md#policies-for-interactive-and-batch-execution). For this feature the [vre-middleware package](https://vre-middleware.readthedocs.io) must be installed. The `WFEngine` object used in the run is dumped into a file in the same folder as the launchpad file. This object can be used independently with the Python API or the GUI of the [vre-middleware package](https://vre-middleware.readthedocs.io). The flag `-q, --qadapter-file` is explained in detail [in this section](#evaluation-of-nodes-of-batch-category the-qadapter-object).
+By default no evaluation is performed. The evaluation with the `-r, --autorun` without the `-a, --async-run` flag is effective only for workflow nodes of [_interactive_ category](resources.md#policies-for-interactive-and-batch-execution). The flag `-a, --async-run` activates background evaluation using a workflow engine with a launcher thread. The evaluation includes workflow nodes of both [_interactive_ and _batch_ categories](resources.md#policies-for-interactive-and-batch-execution). For this feature the [vre-middleware package](https://vre-middleware.readthedocs.io) must be installed. The `WFEngine` object used in the run is dumped into a file in the same folder as the launchpad file. This object can be used independently with the Python API or the GUI of the [vre-middleware package](https://vre-middleware.readthedocs.io) as described in more detail [in this section](#evaluation-of-nodes-of-batch-category). The flag `-q, --qadapter-file` is explained in detail [in this section](#evaluation-of-nodes-of-batch-category).
 By using the `-h, --help` flag a summarized help can be obtained:
@@ -130,8 +129,9 @@ magic commands                     | action
 `%hist`, `%history`                | print the current model source
 `%rerun var1[, var2][, ...]`       | re-evaluate var1, var2, etc.
 `%vary`                            | print the [varied parameters](bulk.md#bulk-processing)
-`%tag`                             | print the [tag section](query.html#the-tag-statement)
+`%tag`                             | print the [tag section](query.md#the-tag-statement)
 `%find <query> [action]`           | perform a global [search](query.md#the-find-command)
+`%help`                            | show this help
 The `%history` command prints all statements (without `print` statements that are not persisted), one per line, with timestamp of most recent evaluation update and current evaluation state. The evaluation state is currently passed through from the FireWorks workflow management system and is described in detail [here](https://materialsproject.github.io/fireworks/reference.html#fireworks-states).
@@ -221,13 +221,21 @@ Cannot convert from 'meter' ([length]) to 'dimensionless' (dimensionless)
 In workflow evaluation mode this error occurs only if the evaluation is enabled and will be issued only if there is `print(b)` statement.
-## Evaluation of nodes of batch category: the _qadapter_ object
+## Evaluation of nodes of batch category
-Once a model has been created in workflow mode, thus added to the database, it can be evaluated using either `texts script` or `texts session` with the `--autorun, -r` flag. In this mode, the workflow nodes of _interactive_ category are evaluated synchronously while the nodes of _batch_ category are not. More information about the workflow node categories can be found [here](resources.md#policies-for-interactive-and-batch-execution). To enable asynchronous background evaluation of nodes of batch category one has to add the `--async-run, -a` flag.
+Once a model has been created in workflow mode, thus added to the database, it can be evaluated using either `texts script` or `texts session` with the `--autorun, -r` flag. In this mode, the workflow nodes of _interactive_ category are evaluated synchronously while the nodes of _batch_ category are not. More information about the workflow node categories can be found [here](resources.md#policies-for-interactive-and-batch-execution). To enable asynchronous background evaluation of nodes of batch category one has to add the `--async-run, -a` flag. As soon as the `texts session` is exited the background evaluation is terminated. This means that no further _batch_ nodes will be scheduled for execution. The already scheduled or running _batch_ nodes will be further processed without interruption or any user actions. The evaluation of a model can be continued either by loading the model in a new session with `texts session -u UUID -r -a` or by running the `wfengine` CLI tool from the [VRE Middware package](https://vre-middleware.readthedocs.io/en/latest/resconfig.html) independently.
+```bash
+wfengine --load-from-file wfengine-<uuid>.yaml -r -c all
+```
+The `wfengine-<uuid>.yaml` file is always created by `texts session` in the same folder as the launchpad file used in the session when the `--async-run, -a` flag is used. The `wfengine-<uuid>.yaml` file contains a dumped `WEngine` object with a complete configuration, including launchpad, model UUID, default qadapter (see below), default worker, etc. Further options for running the `wfengine` tool can be shown with the `--help` flag.
+The evaluation of nodes of batch category happens in an HPC environment governed by a batch system such as SLURM and requires an object called *qadapter*.
 ### Default qadapter
-The evaluation of nodes of batch category happens in an HPC environment governed by a batch system such as SLURM and requires an object called *qadapter*. A default qadapter will be automatically created by the workflow engine and used by the launcher thread. To this end, the default resources in the resource configuration (*resconfig*) object are used. These resources are: default worker, default queue, default launch directory, default group used for accounting, default queue with default computing resources (such as walltime, number of CPUs, amount of memory etc.), default loaded environment modules, default virtual environment. The resconfig object is stored in a resconfig file with default location `$HOME/.fireworks/res_config.yaml` that can be overridden by the environment variable `RESCONFIG_LOC`. If the resconfig file does not exist at the beginning of `texts session`, then it is generated automatically (a prompt will be displayed to accept this step). Further more technical information about res_config can be found [elsewhere](https://vre-middleware.readthedocs.io/en/latest/resconfig.html).
+A default qadapter will be automatically created by the workflow engine and used by the launcher thread. To this end, the default resources in the resource configuration (*resconfig*) object are used. These resources are: default worker, default queue, default launch directory, default group used for accounting, default queue with default computing resources (such as walltime, number of CPUs, amount of memory etc.), default loaded environment modules, default virtual environment. The resconfig object is stored in a resconfig file with default location `$HOME/.fireworks/res_config.yaml` that can be overridden by the environment variable `RESCONFIG_LOC`. If the resconfig file does not exist at the beginning of `texts session`, then it is generated automatically (a prompt will be displayed to accept this step). Further more technical information about res_config can be found [elsewhere](https://vre-middleware.readthedocs.io/en/latest/resconfig.html).
 If necessary, the default qadapter can be overridden by using the optional `--qadapter-file, -q` flag, which takes the name of a qadapter file (e.g. "qadapter.yaml"), containing a qadapter object, as argument. This can be necessary if the default qadapter does not specify some resource needed by the batch job to run. However, the recommended way to fix this is to modify the res_config object instead of providing a default qadapter. If the user has to write a qadapter file then [this tutorial](https://materialsproject.github.io/fireworks/qadapter_programming.html) can be used.

--- a/src/virtmat/language/cli/run_model.py
+++ b/src/virtmat/language/cli/run_model.py
@@ -33,8 +33,6 @@ def add_arguments(parser):
                        help='path to grammar root file', default=GRAMMAR_LOC)
    parser.add_argument('-m', '--mode', required=False, default='instant',
                        help='execution mode', choices=['instant', 'deferred', 'workflow'])
-    parser.add_argument('-t', '--textx-debug', default=False, required=False,
-                        action='store_true', help='enable textx debug mode')
    parser.add_argument('-l', '--launchpad-file', default=LAUNCHPAD_LOC,
                        help='path to launchpad file (workflow mode only)')
    parser.add_argument('-u', '--uuid', default=None, required=False,
@@ -64,15 +62,25 @@ def add_arguments(parser):
 @error_handler
 def run_instant_deferred(clargs, deferred=False, apply_constraints=True):
    """run a model in instant or deferred mode"""
-    meta = metamodel_from_file(clargs.grammar_path, auto_init_attributes=False,
+    def check_clargs(attr_, val_, bstr_):
-                               debug=clargs.textx_debug)
+        if ((val_ in (None, False, True) and getattr(clargs, attr_)) or
+           getattr(clargs, attr_) != val_):
+            warnings.warn(f'argument {attr_} {bstr_}', TextSUserWarning)
+    bstr = f'will be ignored in {clargs.mode} mode'
+    attrs = {'launchpad_file': LAUNCHPAD_LOC, 'uuid': None, 'autorun': False,
+             'qadapter_file': None, 'no_unique_launchdir': False,
+             'no_duplicate_detection': False, 'on_demand': False}
+    for attr, val in attrs.items():
+        check_clargs(attr, val, bstr)
+    meta = metamodel_from_file(clargs.grammar_path, auto_init_attributes=False)
    add_properties(meta, deferred_mode=deferred)
    add_processors(meta, constr_processors=apply_constraints)
    with open(clargs.model_file, 'r', encoding='utf-8') as inp:
        model_str = inp.read()
    return meta.model_from_file(clargs.model_file, deferred_mode=deferred,
-                                source_code=model_str, debug=clargs.textx_debug)
+                                source_code=model_str)
 @error_handler
@@ -97,10 +105,9 @@ def run_workflow(clargs):
    grammar_path = clargs.grammar_path if clargs.uuid is None else None
    session = Session(lp_obj, uuid=clargs.uuid, grammar_path=grammar_path,
                      model_path=clargs.model_file, autorun=clargs.autorun,
-                      on_demand=clargs.on_demand, textx_debug=clargs.textx_debug,
+                      on_demand=clargs.on_demand, detect_duplicates=detect_duplicates,
-                      config_dir=config_dir, qadapter=qadapter,
+                      config_dir=config_dir, qadapter=qadapter, lp_path=lp_path,
-                      unique_launchdir=not clargs.no_unique_launchdir,
+                      unique_launchdir=not clargs.no_unique_launchdir)
-                      detect_duplicates=detect_duplicates, lp_path=lp_path)
    prog = session.get_model()
    session.stop_runner()
    if prog:
@@ -123,29 +130,26 @@ def main(clargs):
        warnings.filterwarnings('ignore')
    warnings.filterwarnings('default', category=TextSUserWarning)
    if clargs.ignore_warnings:
-        warnings.filterwarnings('ignore', category=TextSUserWarning)  # not covered
+        warnings.filterwarnings('ignore', category=TextSUserWarning)
    if clargs.show_model:
+        if clargs.mode != 'instant':
+            msg = 'switching to instant mode due to argument show-model'
+            warnings.warn(msg, TextSUserWarning)
+            clargs.mode = 'instant'
        prog = run_instant_deferred(clargs, apply_constraints=False)
        if prog is not None and not isinstance(prog, str):
            model_export(prog, os.path.splitext(clargs.model_file)[0]+'.dot')
        return
    if clargs.no_interpreter:
+        if clargs.mode != 'instant':
+            msg = 'switching to instant mode due to argument no-interpreter'
+            warnings.warn(msg, TextSUserWarning)
+            clargs.mode = 'instant'
        run_instant_deferred(clargs)
        return
-    if clargs.mode != 'workflow':
-        def check_clargs(attr_, val_, bstr_):
-            if ((val_ in (None, False, True) and getattr(clargs, attr_)) or
-               getattr(clargs, attr_) != val_):
-                warnings.warn(f'argument {attr_} {bstr_}', TextSUserWarning)  # not covered
-        bstr = f'will be ignored in {clargs.mode} mode'
-        attrs = {'launchpad_file': LAUNCHPAD_LOC, 'uuid': None, 'autorun': False,
-                 'qadapter_file': None}
-        for attr, val in attrs.items():
-            check_clargs(attr, val, bstr)
    if clargs.mode == 'instant':
        prog = run_instant_deferred(clargs, deferred=False)
    elif clargs.mode == 'deferred':

--- a/src/virtmat/language/interpreter/session.py
+++ b/src/virtmat/language/interpreter/session.py
@@ -69,7 +69,7 @@ class Session:
    def __init__(self, lpad, uuid=None, grammar_str=None, grammar_path=None,
                 model_str=None, model_path=None, create_new=False, autorun=False,
-                 on_demand=False, async_run=False, textx_debug=False, config_dir=None,
+                 on_demand=False, async_run=False, config_dir=None,
                 detect_duplicates=False, lp_path=None, **wfe_kwargs):
        self.logger = get_logger(__name__)
        if lpad.fw_id_assigner.find_one({}) is None:
@@ -79,10 +79,10 @@ class Session:
        self.lp_path = lp_path
        self.wfe = None
        self.async_run = async_run
-        self.textx_debug = textx_debug
+        self.textx_debug = False
        self.detect_duplicates = detect_duplicates
        self.logger.info('duplicate detection enabled: %s', self.detect_duplicates)
-        self.tx_kwargs = {'auto_init_attributes': False, 'debug': textx_debug}
+        self.tx_kwargs = {'auto_init_attributes': False, 'debug': self.textx_debug}
        if uuid is not None:
            if grammar_str or grammar_path:
                msg = 'provided grammar ignored in favor of grammar from provided uuid'

--- a/src/virtmat/language/interpreter/session_manager.py
+++ b/src/virtmat/language/interpreter/session_manager.py
@@ -271,7 +271,8 @@ class SessionManager(InteractiveConsole):
                   '%rerun var1[, var2][, ...]     re-evaluate var1, var2, etc.\n'
                   '%vary                          print the varied parameters\n'
                   '%tag                           print the tag section\n'
-                   '%find <query> [action]         perform a global search\n')
+                   '%find <query> [action]         perform a global search\n'
+                   '%help                          show this help\n')
            print(msg)
    def is_launcher_running(self):

--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -260,3 +260,29 @@ def test_texts_session_help(lpad_file, _res_config_loc):
    with sp.Popen(command, stdin=sp.PIPE, stdout=sp.PIPE, shell=False) as proc:
        stdout = proc.communicate(input='%help'.encode())[0].decode()
    assert '%exit, %bye, %close, %quit' in stdout
+def test_texts_script_no_warnings(model_file):
+    """test texts script cli tool with no warnings"""
+    command = ['texts', 'script', '-f', model_file, '-r', '-m', 'deferred', '-w']
+    with sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE, shell=False) as proc:
+        assert proc.stdout.read().decode().strip() == 'program output: >>>\n1\n<<<'
+        assert proc.stderr.read().decode() == ''
+def test_texts_script_show_model_deferred_mode(model_file):
+    """test texts script cli tool with the --show-model option in deferred mode"""
+    command = ['texts', 'script', '-f', model_file, '--show-model', '-m', 'deferred']
+    msg = 'Warning: switching to instant mode due to argument show-model'
+    with sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE, shell=False) as proc:
+        assert proc.stdout.read().decode() == ''
+        assert msg in proc.stderr.read().decode()
+def test_texts_script_no_interpreter_deferred_mode(model_file):
+    """test texts script cli tool with the --no-interpreter option in deferred mode"""
+    command = ['texts', 'script', '-f', model_file, '--no-interpreter', '-m', 'deferred']
+    msg = 'Warning: switching to instant mode due to argument no-interpreter'
+    with sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE, shell=False) as proc:
+        assert proc.stdout.read().decode() == ''
+        assert msg in proc.stderr.read().decode()
No results found