How can I create a Jupyter notebook parameter study on the cluster?

I have a jupyter notebook with specified parameters. How can I send notebook to individual nodes with different parameters?

assigned to @ai5116

By rainer.duda on 2022-04-14T11:14:37 (imported from GitLab)

You can create a Python startup script and make use of Papermill in combination with SLURM job arrays.

By rainer.duda on 2022-04-14T11:19:48 (imported from GitLab)

Create a python startup script: my_script.py and fill out the header like:

---

#!/usr/bin/env python

#SBATCH --job-name=your_job_name

#SBATCH --error=%x.%j.err
#SBATCH --output=%x.%j.out

#Here are your parameter variations that you want to test
#SBATCH --array=10,20,50

#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=your-email

#Export your whole virtual environment to the nodes
#SBATCH --export=ALL 

#Specify you wall time
#SBATCH --time=08:00:00

#SBATCH --partition=sdil

#How many GPUs do you need?
#SBATCH --gres=gpu:1

#Now comes the body
import papermill as pm
import os
your_parameter = os.environ.get('SLURM_ARRAY_TASK_ID')
pm.execute_notebook(
'your_notebook.ipynb',
'your_notebook_out_{}.ipynb'.format(your_parameter),
parameters = dict(your_parameter=your_parameter)
)

Start your startup script in the console with: sbatch my_script.py and monitor your jobs with watch squeue

By rainer.duda on 2022-04-19T16:54:09 (imported from GitLab)

closed

reopened

added answered label

added rookie woes label

mentioned in issue #51

By till.riedel on 2022-06-01T16:18:48 (imported from GitLab)

unassigned @till.riedel