Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • kit/virtmat-tools/vre-language
1 result
Show changes
Commits on Source (57)
Hide whitespace changes
Inline Side-by-side
Showing
with 252 additions and 174 deletions
docs/faqs.md
View file @ 5b53e7bc
...@@ -26,4 +26,19 @@ cfg.default_worker.accounts.append(None) ...@@ -26,4 +26,19 @@ cfg.default_worker.accounts.append(None)
cfg.default_worker.default_account = None cfg.default_worker.default_account = None
cfg.to_file(get_resconfig_loc()) cfg.to_file(get_resconfig_loc())
``` ```
Then run `texts` again. Then use the [variable update](scl.md#dealing-with-failures) (using `:=`) to inforce re-evaluation of the resources. A simple `%rerun var` will not be sufficient. Then run `texts` again. Then use the [variable update](scl.md#dealing-with-failures) (using `:=`) to inforce re-evaluation of the resources. A simple `%rerun var` will not be sufficient.
\ No newline at end of file
**Q**: How can I set a datastore configuration alternative to the default `$HOME/.fireworks/datastore_config.yaml`?
**A**: This can be done using the [vre-middleware's](https://vre-middleware.readthedocs.io/en/latest/resconfig.html) Python API. For example, within an iPython command line session:
```python
from virtmat.middleware.resconfig import get_resconfig_loc, get_default_resconfig
cfg = get_default_resconfig()
wcfg = cfg.workers[0] # SEE NOTE
wcfg.envvars.update({'DATASTORE_CONFIG': '/alternative/path_to/my_project/datastore_config.yaml'})
wcfg.default_envvars.append('DATASTORE_CONFIG')
cfg.default_worker = wcfg
cfg.to_file(get_resconfig_loc())
```
**NOTE:** In this example `worker[0]` is assumed as the relevant worker for the datastore configuration. If available, other workers can be selected by using the corresponding index, eg `worker[n]`.
\ No newline at end of file
docs/io.md
View file @ 5b53e7bc
...@@ -70,8 +70,8 @@ In the workflow executor, sometimes a computed parameter value allocates too muc ...@@ -70,8 +70,8 @@ In the workflow executor, sometimes a computed parameter value allocates too muc
If you wish to change these settings you can create a custom datastore configuration file with these contents: If you wish to change these settings you can create a custom datastore configuration file with these contents:
```yaml ```yaml
inline-threshold: 100000 # threshold for offloading data, in bytes
type: file # can be 'gridfs' for database file object storage type: file # can be 'gridfs' for database file object storage
inline-threshold: 100000 # threshold for offloading data, in bytes
path: /path/to/local/workspace # directory used if type is 'file' path: /path/to/local/workspace # directory used if type is 'file'
name: vre_language_datastore # collection name used if type is 'gridfs' name: vre_language_datastore # collection name used if type is 'gridfs'
launchpad: /path/to/launchpad.yaml # path to custom launchpad file used if type is 'gridfs' launchpad: /path/to/launchpad.yaml # path to custom launchpad file used if type is 'gridfs'
...@@ -79,7 +79,8 @@ format: json # 'yaml' and 'hdf5' not implemented ...@@ -79,7 +79,8 @@ format: json # 'yaml' and 'hdf5' not implemented
compress: true # use compression compress: true # use compression
``` ```
The type `file` triggers storage in local files in `path`. The default `path` is `$HOME/.fireworks/vre-language-datastore`. The `path` will be created automatically if it does not exist. The default `launchpad` is `LAUNCHPAD_LOC` as provided by FireWorks. All other default settings are shown in the example above. The `type: file` setting triggers storage in local files in `path`. Setting `type: null` deactivates the mechanism regardless of the other settings.
The default `path` is `$HOME/.fireworks/vre-language-datastore`. The `path` will be created automatically if it does not exist. The default `launchpad` is `LAUNCHPAD_LOC` as provided by FireWorks. All other default settings are shown in the example above.
The default path of the datastore configuration file is `$HOME/.fireworks/datastore_config.yaml`. It will be automatically loaded, if the file exists. If your datastore configuration has a different location then you must set the environment variable The default path of the datastore configuration file is `$HOME/.fireworks/datastore_config.yaml`. It will be automatically loaded, if the file exists. If your datastore configuration has a different location then you must set the environment variable
...@@ -87,7 +88,7 @@ The default path of the datastore configuration file is `$HOME/.fireworks/datast ...@@ -87,7 +88,7 @@ The default path of the datastore configuration file is `$HOME/.fireworks/datast
export DATASTORE_CONFIG=/path/to/datastore_config.yaml export DATASTORE_CONFIG=/path/to/datastore_config.yaml
``` ```
before running the `texts` tool. If the variable `$DATASTORE_CONFIG` should be used then it has to also be added to the `default_envvars` list in the relevant worker as described (here)[https://vre-middleware.readthedocs.io/en/latest/resconfig.html#configuration-of-environment-variables]. before running the `texts` tool. If the variable `$DATASTORE_CONFIG` should be used then it has to also be added to the `default_envvars` list in the relevant worker as described [here](https://vre-middleware.readthedocs.io/en/latest/resconfig.html#configuration-of-environment-variables).
It is not recommended to set `inline-threshold` to very small values, unless you are testing. It is not recommended to set `inline-threshold` to very small values, unless you are testing.
......
docs/scl.md
View file @ 5b53e7bc
...@@ -133,7 +133,7 @@ The file extension (the portion of the path after the `.`) indicates the format ...@@ -133,7 +133,7 @@ The file extension (the portion of the path after the `.`) indicates the format
**NOTE:** If a file with the same name as specified already exists, the export statement will not work, i.e. export allows no file overwriting. **NOTE:** If a file with the same name as specified already exists, the export statement will not work, i.e. export allows no file overwriting.
**NOTE:** While *relative paths*, as in the example above, are supported it is strongly recommended to use *absolute paths*, especially in the workflow evaluation mode. **NOTE:** While *relative paths*, as in the example above, are supported it is strongly recommended to use *absolute paths*, especially in the [workflow evaluation mode](tools.md#workflow-mode).
### Other statements ### Other statements
...@@ -751,6 +751,68 @@ print(sin(2.*pi)) # result: -2.4492935982947064e-16 ...@@ -751,6 +751,68 @@ print(sin(2.*pi)) # result: -2.4492935982947064e-16
print(cos(2.*pi)) # result: 1.0 print(cos(2.*pi)) # result: 1.0
``` ```
## The built-in `info` function
The `info` function takes one argument that can be any parameter. It returns a [Table](#table) with information about the parameter, such as type, datatype (for Series and Arrays), and dimensionality / units if the parameter is numeric and has been evaluated. In [workflow evaluation mode](tools.md#workflow-mode), additional metadata is included in the table when the parameter is a variable.
Example with an expression:
Input:
```
print(info(prop.energy))
```
Output:
```
((name: null),
(type: 'Series'),
(scalar: false),
(numeric: true),
(datatype: 'float'),
(dimensionality: '[mass] * [length] ** 2 / [time] ** 2'),
(units: 'electron_volt'))
```
Example with a variable:
Input:
```
print(info(prop))
```
Output:
```
((name: 'prop'),
(type: 'AMMLProperty'),
(scalar: false),
(numeric: false),
(datatype: null),
('group UUID': 'd9fe0968ed6740888750eb534aababa0'),
('model UUID': '181091fc725242258b3788ed797bf932'),
('node UUID': 'c68c476ef82141d29ae5be555722c27a'),
('node ID': 751),
('parent IDs': (752, 753, 754)),
('node state': 'COMPLETED'),
('created on': '2025-02-25T14:23:32+01:00'),
('updated on': '2025-02-25T14:23:44+01:00'),
('grammar version': 32),
('data schema version': 7),
('python version': '3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]'),
(category: 'interactive'),
(fworker: null),
(dupefinder: true),
('reservation ID': null),
('number of launches': 1),
('number of archived launches': 0),
(launch_dir: ('/mnt/data/ubuntu/work/vre-language/examples/launcher_2025-02-25-13-23-33-167195',)),
('archived launch_dir': ()),
(runtime_secs: 3.495358),
('runtime_secs total': 3.495358))
```
The output of `info` cannot be used as parameter of a variable and therefore is not persistent. Currently, `info` can only be used in [print statements](#the-print-statement) but **does not** trigger [on-demand evaluation](io.md#evaluate-on-demand) of the referenced variables.
The `info` function replaces the deprecated `type` function (grammar version 32 or newer). The `type` function in older compatible grammar versions produces the same output as `info`.
## Loading parameters from file or URL ## Loading parameters from file or URL
Some parameters have the optional syntax allowing to load them from file or download from a URL. The common syntax is Some parameters have the optional syntax allowing to load them from file or download from a URL. The common syntax is
...@@ -761,7 +823,7 @@ Some parameters have the optional syntax allowing to load them from file or down ...@@ -761,7 +823,7 @@ Some parameters have the optional syntax allowing to load them from file or down
Current list of parameters supporting this syntax: `Quantity`, `Bool`, `String`, `Series`, `Table`, `BoolArray`, `StrArray`, `IntArray`, `FloatArray` and `ComplexArray`. Additionally, there are domain-specific parameters that also support this syntax. Current list of parameters supporting this syntax: `Quantity`, `Bool`, `String`, `Series`, `Table`, `BoolArray`, `StrArray`, `IntArray`, `FloatArray` and `ComplexArray`. Additionally, there are domain-specific parameters that also support this syntax.
The `<path>` is a string that must contain the path to the input data file. While *relative paths* are supported it is strongly recommended to use *absolute paths*, especially in the workflow evaluation mode. By default the internal serialization format (in JSON) is used and the file type may be JSON (filename extension `json`) or YAML (file name extensions `yml` or `yaml`). Domain-specific parameters may support further domain-specific formats. The `<path>` is a string that must contain the path to the input data file. While *relative paths* are supported it is strongly recommended to use *absolute paths*, especially in the [workflow evaluation mode](tools.md#workflow-mode). By default the internal serialization format (in JSON) is used and the file type may be JSON (filename extension `json`) or YAML (file name extensions `yml` or `yaml`). Domain-specific parameters may support further domain-specific formats.
## Dealing with missing data or default values ## Dealing with missing data or default values
......
docs/tools.md
View file @ 5b53e7bc
...@@ -2,7 +2,7 @@ ...@@ -2,7 +2,7 @@
To enable the usage of the SCL and AMML, a minimum set of supporting tools is necessary. These tools include a parser, a correctness checker, an interpreter, and a syntax highlighter. All these tools (or subsets of them) can be used via the command-line interface or the Jupyter/iPython kernel. To enable the usage of the SCL and AMML, a minimum set of supporting tools is necessary. These tools include a parser, a correctness checker, an interpreter, and a syntax highlighter. All these tools (or subsets of them) can be used via the command-line interface or the Jupyter/iPython kernel.
## Using the command-line interface ## Using the command-line interface (CLI)
The command-line interface is invoked with the command `texts`. The command-line interface is invoked with the command `texts`.
...@@ -63,6 +63,11 @@ A persistent model can be retrieved in workflow mode by using the flag `-u UUID, ...@@ -63,6 +63,11 @@ A persistent model can be retrieved in workflow mode by using the flag `-u UUID,
By default, in workflow mode every statement will be evaluated in a unique launch directory. Because nodes, that are evaluated in parallel and that share a launch directory, may interfere due to possible local file I/O operations. The default use of unique launch directories can be deactivated for [_interactive_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes by the flag `--no-unique-launchdir`. In this case, one has to make sure that statements giving rise to _interactive_ workflow nodes do not cause conflicting file I/O operations. Note that [_batch_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes are always evaluated in unique launch directories. By default, in workflow mode every statement will be evaluated in a unique launch directory. Because nodes, that are evaluated in parallel and that share a launch directory, may interfere due to possible local file I/O operations. The default use of unique launch directories can be deactivated for [_interactive_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes by the flag `--no-unique-launchdir`. In this case, one has to make sure that statements giving rise to _interactive_ workflow nodes do not cause conflicting file I/O operations. Note that [_batch_](resources.md#policies-for-interactive-and-batch-execution) workflow nodes are always evaluated in unique launch directories.
#### Turning the script into an executable
A script can be turned into an executable by adding `#!/usr/bin/env -S texts script -f` to the top (first line) of the script. Then the file must be given execute permissions by running `chmod +x my_script.vm` and placed somewhere in the search path (defined in `$PATH`). After this, the script can be started as a normal executable by just running `my_script.vm`.
### Interactive session ### Interactive session
The interactive session, started with `texts session`, is a tool managing a session for processing a model in *workflow mode*. Some of the startup flags are the same as for `texts script`. The interactive session, started with `texts session`, is a tool managing a session for processing a model in *workflow mode*. Some of the startup flags are the same as for `texts script`.
...@@ -128,6 +133,7 @@ magic commands | action ...@@ -128,6 +133,7 @@ magic commands | action
`%new` | create a new session and a new model `%new` | create a new session and a new model
`%hist`, `%history` | print the current model source `%hist`, `%history` | print the current model source
`%rerun var1[, var2][, ...]` | re-evaluate var1, var2, etc. `%rerun var1[, var2][, ...]` | re-evaluate var1, var2, etc.
`%cancel var1[, var2][, ...]` | cancel evaluation of var1, var2, etc.
`%vary` | print the [varied parameters](bulk.md#bulk-processing) `%vary` | print the [varied parameters](bulk.md#bulk-processing)
`%tag` | print the [tag section](query.md#the-tag-statement) `%tag` | print the [tag section](query.md#the-tag-statement)
`%find <query> [action]` | perform a global [search](query.md#the-find-command) `%find <query> [action]` | perform a global [search](query.md#the-find-command)
...@@ -137,6 +143,30 @@ The `%history` command prints all statements (without `print` statements that ar ...@@ -137,6 +143,30 @@ The `%history` command prints all statements (without `print` statements that ar
The `%rerun var` command can be used to re-evaluate the parameter of a variable `var` and its descendants, i.e. all parameters containing references to `var`. This is useful when there was some error during evaluation of `var` that was not due to the provided input to `var` but due to side effects, for example there was a computing node or file system failure, network or power outage etc. The `%rerun var` command can be used to re-evaluate the parameter of a variable `var` and its descendants, i.e. all parameters containing references to `var`. This is useful when there was some error during evaluation of `var` that was not due to the provided input to `var` but due to side effects, for example there was a computing node or file system failure, network or power outage etc.
The `%cancel var` command can be used to suspend the evaluation of the parameter of a variable `var`. Depending on the state, the final state of the variable will be different. The possible uses of the `%rerun` and `%cancel` magics, and the [variable update operator](scl.md#dealing-with-failures) (`:=`) are summarized in the following table:
original state | final state | magic / operator
----------------|--------------|--------------------
ARCHIVED | - | -
FIZZLED | WAITING | `%rerun`, `:=`
FIZZLED | DEFUSED | `%cancel`
DEFUSED | WAITING | `%rerun`, `:=`
PAUSED | WAITING | `%rerun`, `:=`
PAUSED | DEFUSED | `%cancel`
WAITING | WAITING | `:=`
WAITING | PAUSED | `%cancel`
READY | READY | `:=`
READY | PAUSED | `%cancel`
RESERVED | WAITING | `%rerun`
RESERVED | PAUSED | `%cancel`
RUNNING | WAITING | `%rerun`
RUNNING | DEFUSED | `%cancel`
COMPLETED | WAITING | `%rerun`, `:=`
**Note**: If the final state is WAITING and all referenced variables are in COMPLETED states then the final state is automatically changed to READY.
**Note**: Variables in ARCHIVED state cannot be updated.
### Duplicate detection ### Duplicate detection
Both *script* and *session* modes have a duplicate detection feature to identify variables in other persistent models that have identical names, parameters and values. This feature ensures safe reuse of such variables and helps to avoid unnecessary repeated evaluations. The duplicate detection can be used in any model but particularly useful in the cases of [sub-model reuse](submodel.md) and of [bulk processing](bulk.md). Both *script* and *session* modes have a duplicate detection feature to identify variables in other persistent models that have identical names, parameters and values. This feature ensures safe reuse of such variables and helps to avoid unnecessary repeated evaluations. The duplicate detection can be used in any model but particularly useful in the cases of [sub-model reuse](submodel.md) and of [bulk processing](bulk.md).
......
examples/builtins.vm 100644 → 100755
View file @ 5b53e7bc
#!/usr/bin/env -S texts script -f
""" """
This is a test set for the scientific computing language This is a test set for the scientific computing language
""" """
......
examples/tuple_iterable.vm
View file @ 5b53e7bc
tup = (1, true, 'abc') tup = (1, true, 'abc')
print(type((1, true, 'abc'))) print(info((1, true, 'abc')))
print(type(tup)) print(info(tup))
print(type(tup[0])) print(info(tup[0]))
print(type(tup[1])) print(info(tup[1]))
print(type(tup[2])) print(info(tup[2]))
print((tup[1], tup[0], tup[2])) print((tup[1], tup[0], tup[2]))
tup2 = ((2, 3), false) tup2 = ((2, 3), false)
......
examples/type_function.vm
View file @ 5b53e7bc
b = 2 b = 2
a = b * 0.2 [m] a = b * 0.2 [m]
c = a * b c = a * b
print(c, type(c)) print(c, info(c))
abcstr = 'abc' abcstr = 'abc'
print(a, type(a)) print(a, info(a))
print(true, type(true)) print(true, info(true))
print(abcstr, type(abcstr)) print(abcstr, info(abcstr))
print(3*5, type(3*5)) print(3*5, info(3*5))
print((float: 1., 2., 3.), type((float: 1., 2., 3.))) print((float: 1., 2., 3.), info((float: 1., 2., 3.)))
print((int: 1, 2, 3), type((int: 1, 2, 3))) print((int: 1, 2, 3), info((int: 1, 2, 3)))
print((bool: true, false), type((bool: true, false))) print((bool: true, false), info((bool: true, false)))
requirements.txt
View file @ 5b53e7bc
...@@ -9,7 +9,6 @@ pyyaml ...@@ -9,7 +9,6 @@ pyyaml
dill dill
ase >=3.23.0 ase >=3.23.0
seaborn seaborn
vre-middleware >=1.2.2 vre-middleware >=1.2.4
jupyter_client jupyter_client
ipykernel ipykernel
pympler
setup.cfg
View file @ 5b53e7bc
...@@ -54,10 +54,9 @@ install_requires = ...@@ -54,10 +54,9 @@ install_requires =
dill dill
ase >=3.23.0 ase >=3.23.0
seaborn seaborn
vre-middleware >=1.2.2 vre-middleware >=1.2.4
jupyter_client jupyter_client
ipykernel ipykernel
pympler
[options.extras_require] [options.extras_require]
test = test =
......
src/virtmat/language/cli/__init__.py
View file @ 5b53e7bc
...@@ -36,3 +36,7 @@ def texts(): ...@@ -36,3 +36,7 @@ def texts():
parser_b.set_defaults(func=run_model.main) parser_b.set_defaults(func=run_model.main)
clargs = parser.parse_args() clargs = parser.parse_args()
clargs.func(clargs) clargs.func(clargs)
# python -m cProfile src/virtmat/language/cli/__init__.py [texts options] 2>&1 | tee cProfile.out
# uncomment for profiling
# if __name__ == '__main__':
# texts()
src/virtmat/language/constraints/typechecks.py
View file @ 5b53e7bc
...@@ -332,12 +332,15 @@ def array_type(self): ...@@ -332,12 +332,15 @@ def array_type(self):
def get_array_type(datatype, typespec): def get_array_type(datatype, typespec):
"""construct and return the proper array type depending on datatype""" """construct and return the proper array type depending on datatype"""
if datatype is None:
return None
try: try:
mtype = next(m for m, d in dtypemap.items() if datatype and issubclass(datatype, d)) mtype = next(m for m, d in dtypemap.items() if issubclass(datatype, d))
except StopIteration as err: except StopIteration as err:
if hasattr(datatype, 'datatype'): if is_array_type(datatype) and hasattr(datatype, 'datatype'):
return get_array_type(datatype.datatype, typespec) return get_array_type(datatype.datatype, typespec)
raise err msg = 'array datatype must be numeric, boolean, string or array'
raise StaticTypeError(msg) from err
typespec['arraytype'] = True typespec['arraytype'] = True
return DType(mtype, (typemap[mtype],), typespec) return DType(mtype, (typemap[mtype],), typespec)
......
src/virtmat/language/grammar/identifiers.tx
View file @ 5b53e7bc
...@@ -9,7 +9,7 @@ TypeKeyword: ...@@ -9,7 +9,7 @@ TypeKeyword:
; ;
FunctionKeyword: FunctionKeyword:
/\b(print|view|vary|if|real|imag|all|any|sum|range|map|filter|reduce|type|tag)\b/ /\b(print|view|vary|if|real|imag|all|any|sum|range|map|filter|reduce|info|tag)\b/
; ;
ExpressionKeyword: ExpressionKeyword:
......
src/virtmat/language/grammar/session.tx
View file @ 5b53e7bc
...@@ -16,6 +16,7 @@ Magic: ...@@ -16,6 +16,7 @@ Magic:
( com = 'uuid' ( arg = /[a-f\d]{32}\b/ )? ) | ( com = 'uuid' ( arg = /[a-f\d]{32}\b/ )? ) |
( com = 'sleep' ( arg = INT )? ) | ( com = 'sleep' ( arg = INT )? ) |
( com = 'rerun' args += ID[','] ) | ( com = 'rerun' args += ID[','] ) |
( com = 'cancel' args += ID[','] ) |
( com = 'find' arg = AltTable load_one ?= 'load one' ) ( com = 'find' arg = AltTable load_one ?= 'load one' )
) )
; ;
......
src/virtmat/language/grammar/virtmat.tx
View file @ 5b53e7bc
/* grammar for virtmat modeling language */ /* grammar for virtmat modeling language */
/* grammar version 31 */ /* grammar version 32 */
import imports import imports
import identifiers import identifiers
...@@ -387,10 +387,10 @@ PrintParameter: ...@@ -387,10 +387,10 @@ PrintParameter:
param = Parameter ( '[' inp_units = Units ']' )? param = Parameter ( '[' inp_units = Units ']' )?
; ;
// type function // info function ('type' keyword is deprecated)
Type: Type:
'type' '(' param = Parameter ')' 'info' '(' param = Parameter ')'
; ;
// view function // view function
......
src/virtmat/language/interpreter/deferred_executor.py
View file @ 5b53e7bc
...@@ -27,9 +27,8 @@ from virtmat.language.utilities.errors import RuntimeTypeError, TEXTX_WRAPPED_EX ...@@ -27,9 +27,8 @@ from virtmat.language.utilities.errors import RuntimeTypeError, TEXTX_WRAPPED_EX
from virtmat.language.utilities.typemap import typemap, checktype, checktype_ from virtmat.language.utilities.typemap import typemap, checktype, checktype_
from virtmat.language.utilities.typemap import is_table_like_type, is_table_like from virtmat.language.utilities.typemap import is_table_like_type, is_table_like
from virtmat.language.utilities.types import is_array, is_scalar, settype from virtmat.language.utilities.types import is_array, is_scalar, settype
from virtmat.language.utilities.types import ScalarNumerical, is_array_type from virtmat.language.utilities.types import ScalarNumerical, get_datatype_name
from virtmat.language.utilities.types import is_scalar_type, is_numeric_type from virtmat.language.utilities.types import is_scalar_type, is_numeric_type
from virtmat.language.utilities.types import get_datatype_name
from virtmat.language.utilities.lists import get_array_aslist from virtmat.language.utilities.lists import get_array_aslist
from virtmat.language.utilities.units import get_units, get_dimensionality from virtmat.language.utilities.units import get_units, get_dimensionality
from virtmat.language.utilities.units import convert_series_units from virtmat.language.utilities.units import convert_series_units
...@@ -333,22 +332,28 @@ def iterable_property_func(self): ...@@ -333,22 +332,28 @@ def iterable_property_func(self):
start_ = self.start start_ = self.start
stop_ = self.stop stop_ = self.stop
step_ = self.step step_ = self.step
array = self.array
def get_sliced_value(value):
return value[start_:stop_:step_] if slice_ else value @settype
def get_sliced_value(*args):
if self.array: value = func(*args)
assert self.obj.type_.datatype is not None if slice_:
if issubclass(self.obj.type_.datatype, str): value = value[start_:stop_:step_]
return (lambda *x: get_sliced_value(func(*x)).values.astype(str), pars) if array:
if issubclass(self.obj.type_.datatype, bool): arr_val = value.values
return (lambda *x: get_sliced_value(func(*x)).values, pars) if isinstance(arr_val, pint_pandas.PintArray):
if issubclass(self.obj.type_.datatype, (int, float, complex)): return arr_val.quantity
return (lambda *x: get_sliced_value(func(*x)).values.quantity, pars) assert isinstance(arr_val, numpy.ndarray)
if is_array_type(self.obj.type_.datatype): if issubclass(arr_val.dtype.type, (numpy.str_, numpy.bool_)):
return (lambda *x: get_nested_array(get_sliced_value(func(*x)).values), pars) return arr_val
return (lambda *x: get_sliced_value(func(*x)).values, pars) if isinstance(arr_val[0], str):
return (settype(lambda *x: get_sliced_value(func(*x))), pars) return arr_val.astype(str)
if is_array(arr_val[0]):
return get_nested_array(arr_val)
msg = 'array datatype must be numeric, boolean, string or array'
raise RuntimeTypeError(msg)
return value
return get_sliced_value, pars
def iterable_query_func(self): def iterable_query_func(self):
......
src/virtmat/language/interpreter/instant_executor.py
View file @ 5b53e7bc
...@@ -21,7 +21,7 @@ from virtmat.language.utilities.errors import InvalidUnitError, RuntimeTypeError ...@@ -21,7 +21,7 @@ from virtmat.language.utilities.errors import InvalidUnitError, RuntimeTypeError
from virtmat.language.utilities.errors import RuntimeValueError from virtmat.language.utilities.errors import RuntimeValueError
from virtmat.language.utilities.typemap import typemap, DType, checktype, checktype_ from virtmat.language.utilities.typemap import typemap, DType, checktype, checktype_
from virtmat.language.utilities.typemap import is_table_like, table_like_type from virtmat.language.utilities.typemap import is_table_like, table_like_type
from virtmat.language.utilities.types import ScalarNumerical, is_array, is_array_type from virtmat.language.utilities.types import ScalarNumerical, is_array
from virtmat.language.utilities.types import is_numeric_type, is_numeric_scalar_type from virtmat.language.utilities.types import is_numeric_type, is_numeric_scalar_type
from virtmat.language.utilities.types import is_scalar_type, is_scalar, settype from virtmat.language.utilities.types import is_scalar_type, is_scalar, settype
from virtmat.language.utilities.types import get_datatype_name from virtmat.language.utilities.types import get_datatype_name
...@@ -294,7 +294,7 @@ def numeric_subarray_value(self): ...@@ -294,7 +294,7 @@ def numeric_subarray_value(self):
@settype @settype
def get_sliced_value(obj): def get_sliced_value(obj):
"""return a value slice of an iterable/sequence data structure object""" """return a slice and/or array of an iterable data structure object"""
value = obj.obj.value value = obj.obj.value
if obj.slice: if obj.slice:
value = value[obj.start:obj.stop:obj.step] value = value[obj.start:obj.stop:obj.step]
...@@ -303,13 +303,14 @@ def get_sliced_value(obj): ...@@ -303,13 +303,14 @@ def get_sliced_value(obj):
if isinstance(array, pint_pandas.PintArray): if isinstance(array, pint_pandas.PintArray):
return array.quantity return array.quantity
assert isinstance(array, numpy.ndarray) assert isinstance(array, numpy.ndarray)
assert obj.obj.type_.datatype is not None if issubclass(array.dtype.type, (numpy.str_, numpy.bool_)):
if is_array_type(obj.obj.type_.datatype): return array
return get_nested_array(array) if isinstance(array[0], str):
if issubclass(obj.obj.type_.datatype, str):
return array.astype(str) return array.astype(str)
assert issubclass(obj.obj.type_.datatype, bool) if is_array(array[0]):
return array return get_nested_array(array)
msg = 'array datatype must be numeric, boolean, string or array'
raise RuntimeTypeError(msg)
return value return value
......
src/virtmat/language/interpreter/session.py
View file @ 5b53e7bc
...@@ -20,10 +20,9 @@ from virtmat.language.utilities.logging import get_logger ...@@ -20,10 +20,9 @@ from virtmat.language.utilities.logging import get_logger
from virtmat.language.utilities.formatters import formatter from virtmat.language.utilities.formatters import formatter
from virtmat.language.utilities.fireworks import get_nodes_info, get_nodes_providing from virtmat.language.utilities.fireworks import get_nodes_info, get_nodes_providing
from virtmat.language.utilities.fireworks import get_ancestors, safe_update from virtmat.language.utilities.fireworks import get_ancestors, safe_update
from virtmat.language.utilities.fireworks import get_model_nodes
from virtmat.language.utilities.serializable import get_serializable from virtmat.language.utilities.serializable import get_serializable
from virtmat.language.utilities.serializable import tag_deserialize
from virtmat.language.utilities.types import get_units from virtmat.language.utilities.types import get_units
from virtmat.language.utilities.mongodb import get_iso_datetime
def update_vary(lpad, uuid, vary): def update_vary(lpad, uuid, vary):
...@@ -157,6 +156,7 @@ class Session: ...@@ -157,6 +156,7 @@ class Session:
uuids = [u for u in self.uuids if u is not None] uuids = [u for u in self.uuids if u is not None]
wf_query = {'metadata.uuid': {'$in': uuids}} wf_query = {'metadata.uuid': {'$in': uuids}}
self.wfe = WFEngine(self.lpad, wf_query=wf_query, **wfe_kwargs) self.wfe = WFEngine(self.lpad, wf_query=wf_query, **wfe_kwargs)
self.logger.info(' Started wfengine.')
if config_dir is not None: if config_dir is not None:
self.wfe.to_file(os.path.join(config_dir, f'wfengine-{uuid4().hex}.yaml')) self.wfe.to_file(os.path.join(config_dir, f'wfengine-{uuid4().hex}.yaml'))
if self.on_demand: if self.on_demand:
...@@ -167,7 +167,9 @@ class Session: ...@@ -167,7 +167,9 @@ class Session:
def start_runner(self): def start_runner(self):
"""a helper function to activate evaluation""" """a helper function to activate evaluation"""
if self.async_run and self.wfe: if self.async_run and self.wfe:
self.wfe.start() if not (self.wfe.thread and self.wfe.thread.is_alive()):
self.wfe.start()
self.logger.info(' Started launcher thread.')
self.autorun = False self.autorun = False
else: else:
self.autorun = True self.autorun = True
...@@ -175,7 +177,9 @@ class Session: ...@@ -175,7 +177,9 @@ class Session:
def stop_runner(self): def stop_runner(self):
"""a helper function to deactivate evaluation""" """a helper function to deactivate evaluation"""
if self.async_run and self.wfe: if self.async_run and self.wfe:
self.wfe.stop(join=True) if self.wfe.thread and self.wfe.thread.is_alive():
self.wfe.stop(join=True)
self.logger.info(' Stopped launcher thread.')
else: else:
self.autorun = False self.autorun = False
...@@ -184,51 +188,13 @@ class Session: ...@@ -184,51 +188,13 @@ class Session:
add_properties(self.metamodel, deferred_mode=True, workflow_mode=True) add_properties(self.metamodel, deferred_mode=True, workflow_mode=True)
add_processors(self.metamodel, wflow_processors=True) add_processors(self.metamodel, wflow_processors=True)
def get_model_nodes(self, uuid):
"""return the list of nodes in a persistent model with provided uuid"""
wf_q = {'metadata.uuid': uuid}
fw_q = {'spec._source_code': {'$exists': True}}
fw_p = {'spec._source_code': True, 'updated_on': True, 'state': True}
wfs = get_nodes_info(self.lpad, wf_q, fw_q, fw_p)
if wfs:
return wfs[0]['nodes']
return []
def get_model_str(self, uuid, model_str=''): def get_model_str(self, uuid, model_str=''):
"""updates the persistent model string with model_str""" """updates the persistent model string with model_str"""
fws = self.get_model_nodes(uuid) fws = get_model_nodes(self.lpad, uuid)
lines = [line for fwk in fws for line in fwk['spec']['_source_code']] lines = [line for fwk in fws for line in fwk['spec']['_source_code']]
pers_model_str = ';'.join(filter(None, list(set(lines)))) pers_model_str = ';'.join(filter(None, list(set(lines))))
return ';'.join(filter(None, (pers_model_str, model_str))) return ';'.join(filter(None, (pers_model_str, model_str)))
def get_model_history(self, uuid):
"""return a formatted node history with some node attributes"""
dct = {'state': [], 'updated_on': [], 'source': []}
for fwk in self.get_model_nodes(uuid):
if fwk['spec']['_source_code']:
dct['state'].append(fwk['state'])
dct['updated_on'].append(get_iso_datetime(fwk['updated_on']))
dct['source'].append(fwk['spec']['_source_code'])
df = pandas.DataFrame(dct).sort_values('updated_on').sort_values('state')
df = df[['state', 'updated_on', 'source']]
lines = []
for state, time, src in df.itertuples(index=False, name=None):
for line in src:
line = line.replace('\n', '\n' + ' '*37)
lines.append(' '.join((f'{state:10}', time, line)))
return '\n'.join(lines)
def get_model_tag(self):
"""return the model tag as pandas dataframe"""
wf_q = {'metadata.uuid': self.uuid}
fw_q = {'name': '_fw_meta_node'}
fw_p = {'spec._tag': True}
wfs = get_nodes_info(self.lpad, wf_q, fw_q, fw_p)
assert len(wfs) == 1
assert len(wfs[0]['nodes']) == 1
tag_dct = wfs[0]['nodes'][0]['spec']['_tag']
return tag_deserialize(tag_dct) if tag_dct else None
def process_models(self, model_str=None, model_path=None, active_uuid=None): def process_models(self, model_str=None, model_path=None, active_uuid=None):
"""process all models in the group with provided single input""" """process all models in the group with provided single input"""
strns, paths, varies = self._process_strings_paths(model_str, model_path) strns, paths, varies = self._process_strings_paths(model_str, model_path)
...@@ -538,7 +504,7 @@ class Session: ...@@ -538,7 +504,7 @@ class Session:
if textx_isinstance(obj, meta['Vary']): if textx_isinstance(obj, meta['Vary']):
vary_statements.append(obj) vary_statements.append(obj)
if vary_statements: if vary_statements:
fws = self.get_model_nodes(self.uuid) fws = get_model_nodes(self.lpad, self.uuid)
pers_model_lst = [s for f in fws for s in f['spec']['_source_code']] pers_model_lst = [s for f in fws for s in f['spec']['_source_code']]
statements = [] statements = []
for obj in model.statements: for obj in model.statements:
...@@ -591,5 +557,8 @@ class Session: ...@@ -591,5 +557,8 @@ class Session:
if not textx_isinstance(model, self.metamodel['Program']): if not textx_isinstance(model, self.metamodel['Program']):
model = None model = None
elif self.wfe and self.on_demand and autorun_od: elif self.wfe and self.on_demand and autorun_od:
self.wfe.nodes_torun = list(set(self.wfe.nodes_torun+model.fw_ids_torun)) self.logger.info(' Setting nodes to run on demand.')
nodes_to_run = list(set(self.wfe.nodes_torun+model.fw_ids_torun))
self.logger.debug(' Nodes to run: %s', nodes_to_run)
self.wfe.nodes_torun = nodes_to_run
return model, getattr(model, 'uuid', uuid) return model, getattr(model, 'uuid', uuid)
src/virtmat/language/interpreter/session_manager.py
View file @ 5b53e7bc
"""manage sessions for dynamic model processing / incremental development""" """manage sessions for dynamic model processing / incremental development"""
import os import os
import sys import sys
import importlib
import readline import readline
from code import InteractiveConsole from code import InteractiveConsole
from functools import cached_property from functools import cached_property
...@@ -9,15 +10,16 @@ from virtmat.language.metamodel.processors import table_processor, null_processo ...@@ -9,15 +10,16 @@ from virtmat.language.metamodel.processors import table_processor, null_processo
from virtmat.language.utilities.textx import GRAMMAR_LOC, TextXCompleter, GrammarString from virtmat.language.utilities.textx import GRAMMAR_LOC, TextXCompleter, GrammarString
from virtmat.language.utilities.textx import get_identifiers from virtmat.language.utilities.textx import get_identifiers
from virtmat.language.utilities.errors import error_handler, ModelNotFoundError from virtmat.language.utilities.errors import error_handler, ModelNotFoundError
from virtmat.language.utilities.errors import textxerror_wrap, UpdateError from virtmat.language.utilities.errors import textxerror_wrap
from virtmat.language.utilities.errors import RuntimeTypeError, QueryError from virtmat.language.utilities.errors import RuntimeTypeError, QueryError
from virtmat.language.utilities.logging import get_logger from virtmat.language.utilities.logging import get_logger
from virtmat.language.utilities.warnings import warnings, TextSUserWarning from virtmat.language.utilities.warnings import warnings, TextSUserWarning
from virtmat.language.utilities.formatters import formatter from virtmat.language.utilities.formatters import formatter
from virtmat.language.utilities.fireworks import get_nodes_providing from virtmat.language.utilities.fireworks import get_model_history, get_models_overview
from virtmat.language.utilities.fireworks import rerun_vars, cancel_vars, get_model_tag
from virtmat.language.utilities.fireworks import get_var_names
from virtmat.language.utilities.serializable import tag_serialize from virtmat.language.utilities.serializable import tag_serialize
from virtmat.language.utilities.typemap import checktype from virtmat.language.utilities.typemap import checktype
from virtmat.language.utilities.mongodb import get_iso_datetime
from virtmat.language.constraints.typechecks import (tuple_type, series_type, table_type, from virtmat.language.constraints.typechecks import (tuple_type, series_type, table_type,
dict_type, array_type, quantity_type, dict_type, array_type, quantity_type,
alt_table_type) alt_table_type)
...@@ -98,11 +100,26 @@ def expand_query_prefix(query): ...@@ -98,11 +100,26 @@ def expand_query_prefix(query):
return {k: _recursive_q(v, p_map[k]) for k, v in query.items()} return {k: _recursive_q(v, p_map[k]) for k, v in query.items()}
def get_prettytable(dataframe):
"""convert pandas dataframe to a prettytable object"""
try:
module = importlib.import_module('prettytable')
class_ = getattr(module, 'PrettyTable')
except (ImportError, AttributeError):
return str(dataframe) # failover, not covered
table = class_(list(dataframe.columns))
for tpl in dataframe.itertuples(index=False, name=None):
table.add_row(tpl)
table.align = 'l'
table.max_width = 120
return table
class SessionManager(InteractiveConsole): class SessionManager(InteractiveConsole):
"""session manager for basic interactive work using a text terminal""" """session manager for basic interactive work using a text terminal"""
options = ['%exit', '%bye', '%close', '%quit', '%start', '%stop', '%new', options = ['%exit', '%bye', '%close', '%quit', '%start', '%stop', '%new',
'%vary', '%history', '%hist', '%tag', '%uuid', '%sleep', '%vary', '%history', '%hist', '%tag', '%uuid', '%sleep', '%cancel',
'%rerun', '%find', '%help', 'use', 'print', 'view', 'vary', 'tag'] '%rerun', '%find', '%help', 'use', 'print', 'view', 'vary', 'tag']
def __init__(self, lpad, **kwargs): def __init__(self, lpad, **kwargs):
...@@ -233,9 +250,14 @@ class SessionManager(InteractiveConsole): ...@@ -233,9 +250,14 @@ class SessionManager(InteractiveConsole):
elif model.com == 'vary': elif model.com == 'vary':
print('vary:', formatter(self.session.get_vary_df())) print('vary:', formatter(self.session.get_vary_df()))
elif model.com in ('hist', 'history'): elif model.com in ('hist', 'history'):
print(self.session.get_model_history(self.uuid)) print(get_prettytable(get_model_history(self.lpad, self.uuid)))
unres = self.session.wfe and self.session.wfe.get_unreserved_nodes() or []
lostj = self.session.wfe and self.session.wfe.get_lost_jobs() or []
unres_vars = unres and get_var_names(self.lpad, unres)
lostj_vars = lostj and get_var_names(self.lpad, lostj)
print(f'Unreserved: {unres_vars}, Lost: {lostj_vars}' if unres or lostj else '')
elif model.com == 'tag': elif model.com == 'tag':
print(formatter(self.session.get_model_tag())) print(formatter(get_model_tag(self.lpad, self.uuid)))
elif model.com == 'find': elif model.com == 'find':
get_logger(__name__).debug('process_magic: find: %s', formatter(model.arg.value)) get_logger(__name__).debug('process_magic: find: %s', formatter(model.arg.value))
try: try:
...@@ -244,26 +266,18 @@ class SessionManager(InteractiveConsole): ...@@ -244,26 +266,18 @@ class SessionManager(InteractiveConsole):
get_logger(__name__).error('process_magic: %s', str(err)) get_logger(__name__).error('process_magic: %s', str(err))
raise QueryError(err) from err raise QueryError(err) from err
get_logger(__name__).debug('process_magic: query: %s', q_dict) get_logger(__name__).debug('process_magic: query: %s', q_dict)
wfs = self.get_wflows_from_user_query(q_dict) matching_uuids = self.get_wflows_from_user_query(q_dict)
if model.load_one: if matching_uuids:
wf_uuids = [wf['metadata']['uuid'] for wf in wfs] if model.load_one:
if wf_uuids and self.uuid not in wf_uuids: if self.uuid not in matching_uuids:
self.switch_model(wf_uuids[0]) self.switch_model(matching_uuids[0])
print('uuids:', formatter(self.uuid), formatter(self.session.uuids)) print('uuids:', formatter(self.uuid), formatter(self.session.uuids))
elif wfs: else:
lines = [] print(get_prettytable(get_models_overview(self.lpad, matching_uuids)))
for wf in wfs:
lines.append(' '.join((f'{wf["state"]:10}',
get_iso_datetime(wf['updated_on']),
wf['metadata']['uuid'])))
print('\n'.join(lines))
elif model.com == 'rerun': elif model.com == 'rerun':
for name in model.args: rerun_vars(self.lpad, self.uuid, model.args)
fw_ids = get_nodes_providing(self.lpad, self.uuid, name) elif model.com == 'cancel':
if len(fw_ids) == 0: cancel_vars(self.lpad, self.uuid, model.args)
raise UpdateError(f'Variable {name} not found in the model.')
assert len(fw_ids) == 1
self.lpad.rerun_fw(fw_ids[0])
else: else:
assert model.com == 'help' assert model.com == 'help'
msg = ('%exit, %bye, %close, %quit close (quit) the session\n' msg = ('%exit, %bye, %close, %quit close (quit) the session\n'
...@@ -275,6 +289,7 @@ class SessionManager(InteractiveConsole): ...@@ -275,6 +289,7 @@ class SessionManager(InteractiveConsole):
'%new create a new session and a new model\n' '%new create a new session and a new model\n'
'%hist, %history print the current model source\n' '%hist, %history print the current model source\n'
'%rerun var1[, var2][, ...] re-evaluate var1, var2, etc.\n' '%rerun var1[, var2][, ...] re-evaluate var1, var2, etc.\n'
'%cancel var1[, var2][, ...] cancel evaluation of var1, var2, etc.\n'
'%vary print the varied parameters\n' '%vary print the varied parameters\n'
'%tag print the tag section\n' '%tag print the tag section\n'
'%find <query> [action] perform a global search\n' '%find <query> [action] perform a global search\n'
...@@ -294,8 +309,9 @@ class SessionManager(InteractiveConsole): ...@@ -294,8 +309,9 @@ class SessionManager(InteractiveConsole):
wf_ids = self.lpad.get_fw_ids_in_wfs(q_dict.get('meta'), fw_query=fw_q) wf_ids = self.lpad.get_fw_ids_in_wfs(q_dict.get('meta'), fw_query=fw_q)
if q_data: if q_data:
wf_ids = self.lpad.get_fw_ids_in_wfs({'nodes': {'$in': wf_ids}}, q_data) wf_ids = self.lpad.get_fw_ids_in_wfs({'nodes': {'$in': wf_ids}}, q_data)
wf_p = {'metadata.uuid': True, 'state': True, 'updated_on': True} wf_q = {'nodes': {'$in': wf_ids}}
return list(self.lpad.workflows.find({'nodes': {'$in': wf_ids}}, wf_p)) wf_p = {'metadata.uuid': True}
return [w['metadata']['uuid'] for w in self.lpad.workflows.find(wf_q, wf_p)]
def switch_model(self, new_uuid): def switch_model(self, new_uuid):
"""switch from one to another model""" """switch from one to another model"""
......
src/virtmat/language/interpreter/workflow_executor.py
View file @ 5b53e7bc
...@@ -23,6 +23,9 @@ from virtmat.language.utilities.firetasks import FunctionTask, ExportDataTask, S ...@@ -23,6 +23,9 @@ from virtmat.language.utilities.firetasks import FunctionTask, ExportDataTask, S
from virtmat.language.utilities.fireworks import run_fireworks, get_ancestors from virtmat.language.utilities.fireworks import run_fireworks, get_ancestors
from virtmat.language.utilities.fireworks import get_nodes_providing, get_parent_nodes from virtmat.language.utilities.fireworks import get_nodes_providing, get_parent_nodes
from virtmat.language.utilities.fireworks import safe_update, get_nodes_info, retrieve_value from virtmat.language.utilities.fireworks import safe_update, get_nodes_info, retrieve_value
from virtmat.language.utilities.fireworks import get_fw_metadata, get_launches
from virtmat.language.utilities.fireworks import get_representative_launch
from virtmat.language.utilities.fireworks import append_wf
from virtmat.language.utilities.serializable import DATA_SCHEMA_VERSION from virtmat.language.utilities.serializable import DATA_SCHEMA_VERSION
from virtmat.language.utilities.serializable import FWDataObject, tag_serialize from virtmat.language.utilities.serializable import FWDataObject, tag_serialize
from virtmat.language.utilities.textx import isinstance_m, get_identifiers from virtmat.language.utilities.textx import isinstance_m, get_identifiers
...@@ -36,7 +39,6 @@ from virtmat.language.utilities.types import is_numeric, is_numeric_type ...@@ -36,7 +39,6 @@ from virtmat.language.utilities.types import is_numeric, is_numeric_type
from virtmat.language.utilities.types import get_datatype_name from virtmat.language.utilities.types import get_datatype_name
from virtmat.language.utilities.units import get_units, get_dimensionality from virtmat.language.utilities.units import get_units, get_dimensionality
from virtmat.language.utilities.logging import get_logger from virtmat.language.utilities.logging import get_logger
from virtmat.language.utilities.mongodb import get_iso_datetime
from virtmat.language.utilities.compatibility import get_grammar_version from virtmat.language.utilities.compatibility import get_grammar_version
from .instant_executor import program_value, plain_type_value from .instant_executor import program_value, plain_type_value
...@@ -230,7 +232,7 @@ def variable_value(self): ...@@ -230,7 +232,7 @@ def variable_value(self):
logger = get_logger(__name__) logger = get_logger(__name__)
logger.debug('variable_value:%s', repr(self)) logger.debug('variable_value:%s', repr(self))
model = get_model(self) model = get_model(self)
fw_p = {'state': True, 'fw_id': True, 'launches': True} fw_p = {'state': True, 'fw_id': True, 'launches': True, 'archived_launches': True}
fw_dct = model.lpad.fireworks.find_one({'name': self.__fw_name}, fw_p) fw_dct = model.lpad.fireworks.find_one({'name': self.__fw_name}, fw_p)
if fw_dct['state'] == 'COMPLETED': if fw_dct['state'] == 'COMPLETED':
return retrieve_value(model.lpad, fw_dct['launches'][-1], self.name) return retrieve_value(model.lpad, fw_dct['launches'][-1], self.name)
...@@ -251,8 +253,7 @@ def variable_value(self): ...@@ -251,8 +253,7 @@ def variable_value(self):
raise EvaluationError(''.join(lst)) from exc raise EvaluationError(''.join(lst)) from exc
raise EvaluationError('state FIZZLED but no exception found') # not covered raise EvaluationError('state FIZZLED but no exception found') # not covered
if fw_dct['state'] == 'WAITING': if fw_dct['state'] == 'WAITING':
parents = get_ancestors(model.lpad, fw_dct['fw_id']) fw_q = {'fw_id': {'$in': get_ancestors(model.lpad, fw_dct['fw_id'])}, 'state': 'FIZZLED'}
fw_q = {'fw_id': {'$in': parents}, 'state': 'FIZZLED'}
fizzled = list(model.lpad.fireworks.find(fw_q, projection={'name': True})) fizzled = list(model.lpad.fireworks.find(fw_q, projection={'name': True}))
if fizzled: if fizzled:
msg = f'Evaluation of {self.name} not possible due to failed ancestors: ' msg = f'Evaluation of {self.name} not possible due to failed ancestors: '
...@@ -268,10 +269,13 @@ def variable_value(self): ...@@ -268,10 +269,13 @@ def variable_value(self):
logger.error('variable_value:%s ancestor error', repr(self)) logger.error('variable_value:%s ancestor error', repr(self))
raise AncestorEvaluationError(msg) raise AncestorEvaluationError(msg)
raise NonCompletedException raise NonCompletedException
if fw_dct['state'] in ['READY', 'RESERVED', 'RUNNING']: if fw_dct['state'] in ['READY', 'RESERVED', 'RUNNING', 'PAUSED', 'DEFUSED']:
raise NonCompletedException raise NonCompletedException
logger.critical('variable_value:%s unknown node state', repr(self)) # not covered assert fw_dct['state'] == 'ARCHIVED'
raise RuntimeError('Unknown node state') # not covered launch = get_representative_launch(get_launches(model.lpad, fw_dct['archived_launches']))
if launch and launch.state == 'COMPLETED':
return retrieve_value(model.lpad, launch.launch_id, self.name)
raise NonCompletedException
@error_handler @error_handler
...@@ -314,40 +318,8 @@ def type_value(self): ...@@ -314,40 +318,8 @@ def type_value(self):
model = get_model(self) model = get_model(self)
var_list = get_children_of_type('Variable', model) var_list = get_children_of_type('Variable', model)
var = next(v for v in var_list if v.name == par.ref.name) var = next(v for v in var_list if v.name == par.ref.name)
proj_keys = ['fw_id', 'name', 'state', 'created_on', 'updated_on', dct.update(get_fw_metadata(model.lpad, {'metadata.uuid': model.uuid},
'launches', 'archived_launches', 'spec._category', {'name': var.__fw_name}))
'spec._worker', 'spec._grammar_version', 'spec._python_version',
'spec._data_schema_version', 'spec._dupefinder']
wfs = get_nodes_info(model.lpad, {'metadata.uuid': model.uuid},
{'name': var.__fw_name}, {k: True for k in proj_keys},
{'metadata.g_uuid': True, 'parent_links': True})
assert len(wfs) == 1
assert len(wfs[0]['nodes']) == 1
node = wfs[0]['nodes'][0]
launch_q = {'launch_id': {'$in': node['launches']}}
launch_proj = {'launch_dir': True}
launches = model.lpad.launches.find(launch_q, launch_proj)
launch_q = {'launch_id': {'$in': node['archived_launches']}}
arch_launches = model.lpad.launches.find(launch_q, launch_proj)
dct.update({
'group UUID': wfs[0]['metadata']['g_uuid'],
'model UUID': model.uuid,
'node UUID': var.__fw_name,
'node ID': node['fw_id'],
'parent IDs': wfs[0]['parent_links'].get(str(node['fw_id']), []),
'node state': node['state'],
'created on': get_iso_datetime(node['created_on']),
'updated on': get_iso_datetime(node['updated_on']),
'grammar version': node['spec'].get('_grammar_version'),
'data schema version': node['spec'].get('_data_schema_version'),
'python version': node['spec'].get('_python_version'),
'category': node['spec'].get('_category'),
'fworker': node['spec'].get('_worker'),
'dupefinder': bool(node['spec'].get('_dupefinder')),
'number of launches': len(node['launches']),
'number of archived launches': len(node['archived_launches']),
'launch_dir': [lnch['launch_dir'] for lnch in launches],
'archived launch_dir': [lnch['launch_dir'] for lnch in arch_launches]})
return pandas.DataFrame([dct]) return pandas.DataFrame([dct])
...@@ -504,7 +476,7 @@ def append_var_nodes(model): ...@@ -504,7 +476,7 @@ def append_var_nodes(model):
parents = get_parent_nodes(model.lpad, model.uuid, node) parents = get_parent_nodes(model.lpad, model.uuid, node)
if None not in parents: if None not in parents:
get_logger(__name__).debug('appending %s, parents %s', node, parents) get_logger(__name__).debug('appending %s, parents %s', node, parents)
model.lpad.append_wf(Workflow([nodes.pop(ind)]), fw_ids=parents) append_wf(model.lpad, Workflow([nodes.pop(ind)]), fw_ids=parents)
break break
assert len(nodes) < num_nodes assert len(nodes) < num_nodes
logger.debug('appended %s new variable nodes', nodes_len) logger.debug('appended %s new variable nodes', nodes_len)
...@@ -524,9 +496,9 @@ def append_output_nodes(model): ...@@ -524,9 +496,9 @@ def append_output_nodes(model):
if nodes: if nodes:
assert len(nodes) == 1 assert len(nodes) == 1
obj_to.__fw_name = next(n['name'] for n in nodes) obj_to.__fw_name = next(n['name'] for n in nodes)
else: # not covered else:
parents = get_nodes_providing(model.lpad, model.uuid, obj_to.ref.name) parents = get_nodes_providing(model.lpad, model.uuid, obj_to.ref.name)
model.lpad.append_wf(Workflow([obj_to.firework]), fw_ids=parents) append_wf(model.lpad, Workflow([obj_to.firework]), fw_ids=parents)
logger.debug('added output node for var %s', obj_to.ref.name) logger.debug('added output node for var %s', obj_to.ref.name)
......
src/virtmat/language/utilities/ase_params.py
View file @ 5b53e7bc
...@@ -647,7 +647,7 @@ spec = { ...@@ -647,7 +647,7 @@ spec = {
}, },
'eos': { 'eos': {
'default': 'sjeos', 'default': 'sjeos',
'choices': ['sjeos', 'taylor', 'murnaghan', 'birch', 'birchmurnaghan' 'choices': ['sjeos', 'taylor', 'murnaghan', 'birch', 'birchmurnaghan',
'pouriertarantola', 'vinet', 'antonschmidt', 'p3'], 'pouriertarantola', 'vinet', 'antonschmidt', 'p3'],
'type': str, 'type': str,
'units': None, 'units': None,
......

Help | Imprint | Privacy policy | Accessibility | Contact