Asynchronous model execution
The default deferred workflow execution is running synchronously in the same thread and process on the same host as the interpreter. This has to be changed for productive use cases where many of the nodes require longer time for evaluation so that the control is released to the main thread running the session, the parser and the iterpreter. This is not only to unload the main thread but also to enable parallel execution of nodes using the batch system.
To this end, we can try to use the WFEngine
class from the VRE Middlware package. For that we have to set the default worker type: interactive
, batch
, or remote
. Then, the evaluation strategy should be implemented in that package -> feature request issue.
As input for that type of evaluation we need a list of targets to evaluate (similar to a makefile). These and their dependencies will be then evaluated by the launcher thread. The list should be shared between the session and the launcher thread and can be updated asynchronously in the session manager.