How run a notebook via slurm on the command line?

I want to e.g. run my notebook longer that 4h (See also #46)

There are several ways how you can execute notebooks on the cluster / from the commandline:

• NBConvert
• Papermill *Recommended for parameter studies. (see #47)
• nbtoolbelt *For older Pandas Versions

building on #46 you can use e.g. nbconvert:

#SBATCH --mail-user=duda@kit.edu
#SBATCH --export=ALL
#SBATCH --time=01:00:00
#SBATCH --partition=sdil
#SBATCH --gres=gpu:1

import os
import subprocess
import nbconvert
import jupyter
from subprocess import PIPE, STDOUT

p = subprocess.Popen(["jupyter", 
                      "nbconvert",
                      "--to", 
                      "notebook",
                      "--execute", 
                      "HPC_Tensorflow_SN.ipynb"
                      ], stdout=PIPE, stderr=STDOUT, bufsize=0)

This job script can be executed via:

sbatch job_script.py (Make sure that you are in the correct folder! The job script lies within the same folder than the Jupyter notebook)

By till.riedel on 2022-06-01T16:18:48 (imported from GitLab)

mentioned in issue #49

By till.riedel on 2022-06-01T16:18:18 (imported from GitLab)

mentioned in issue #53

By ployplearn.ravivanpong on 2022-07-22T16:38:10 (imported from GitLab)

Add #!/usr/bin/env python

By tim.schneegans on 2023-06-27T15:48:51 (imported from GitLab)