Enable incremental development / dynamic model

This issue can be solved only after issue https://git.scc.kit.edu/virtmat-tools/vre-language/-/issues/23 (deferred execution) is completely solved.

What is "incremental development"?

A model can have a simple life-cycle:

1. write model
2. parse model
3. apply constraints to the model (such as type checks)
4. interpret the model
5. execute the model

The deferred execution and persistence of the model allow a model extension after the last two steps (interpreter and executor). This opens up a mechanism for rapid design using short interactive steps (like Jupyter notebook cells) to incrementally and dynamically extend the model (before, during or after its execution).

We can summarize like this

1. The model is not static but dynamic, i.e. it can be extended before, during and after runtime.
2. Data from the existing model can be re-used at any later time in the model extensions.

Stages of incremental development

Stage 1: Start a model from scratch

0. Create a metamodel from the grammar and generate a parser.
1. Parse a textual model.
2. Create an abstract model consisting of plain Python objects.
3. Run model and object processors (constraints, interpreter)
   • a workflow is created on the database
   • source code is saved (per node)
   • workflow UUID is generated (workflow metadata)
   • the grammar is also saved (workflow metadata)
4. Run interpreter for print() statements, if any - this step is asynchronous

Stage 2: Extend a model

5. Load a model from persistent storage:
   • UUID is specified
   • The grammar is loaded from the database.
   • Persisted model source is loaded from the database
6. Metamodel and parser are generated from grammar
7. New source as added to persisted model source
8. The steps 1-4 from Stage 1 are repeated except for
   • The UUID and the grammar are not created and saved (workflow metadata cannot be changed anyway).
   • The model processor that creates workflow nodes must "know" which objects are already persisted and which are not. There are two ways to implement this: 1) make a query to check if the name exists in the workflow; 2) the source code of the persisted model is annotated by the loader (step 5).

Stage 2 can be repeated as many times as necessary.

Issues

Issue 1: Original or surrogate source code?

The persistent source code is mostly not necessary for a restart - only the names and the types of the variables from the global scope are necessary. For these, meta source code can be created with some special rule in the grammar. For some objects however the actual attributes will be necessary to do evaluations with references for example:

s = (a: 1, 2)
...
series_name = s.name

The the restart phase s will be visible and the type is (int, 2). But the attribute name will not be available.

Another example

tab = ((a: 1, 2), b: (true, false)) # type is ((int, int), 2)
...
c = filter(x: x, tab.b)
d = tab.a where b == true

Here, in the restart stage we need the series attributes of the tab (Table class) object. However, this depends on the implementation of the interpreter (when we call the func property on the ObjectProperty object). In addition, the parser will search for a reference of metamodel type Series that will not exist in the reloaded model.

Rationale: The whole original persisted model source code should be parsed.

Issue 2: Performance issues

Memory usage and parser and interpreter run times. The whole source must be parsed and the model must be re-instantiated at every change. This might become a problem for interactive use (-> Jupyter kernel). To decrease memory usage the model object should be in the scope of a function that is called by the __main__ Python module. Also in a loop in the __main__ module the object will be retired and garbage collected if the same name is used to construct the updated model. The nice solution would be to have a feature in textX to update an existing model.