Skip to content

Commit 7b7b647

Browse files
LisaBockLisaBock
authored
Merge pull request #409 from ESMValGroup/update_epsiode_04
Update episode 4 "Running your first recipe" for v2.14.0
2 parents 669d610 + 39d710f commit 7b7b647

1 file changed

Lines changed: 169 additions & 12 deletions

File tree

‎_episodes/04-recipe.md‎

Lines changed: 169 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: "Running your first recipe"
33
teaching: 15
44
exercises: 15
5-
compatibility: ESMValTool v2.13.0
5+
compatibility: ESMValTool v2.14.0
66

77
questions:
88
- "How to run a recipe?"
@@ -50,12 +50,168 @@ output to the command line. The final message should be "Run was successful".
5050
The exact output varies depending on your machine, but it should look something
5151
like the example log output below.
5252

53-
```
53+
> ## Output
54+
>
55+
> ```txt
56+
>
57+
> 2026-04-02 12:02:54,972 UTC [3235444] INFO
58+
> ______________________________________________________________________
59+
> _____ ____ __ ____ __ _ _____ _
60+
> | ____/ ___|| \/ \ \ / /_ _| |_ _|__ ___ | |
61+
> | _| \___ \| |\/| |\ \ / / _` | | | |/ _ \ / _ \| |
62+
> | |___ ___) | | | | \ V / (_| | | | | (_) | (_) | |
63+
> |_____|____/|_| |_| \_/ \__,_|_| |_|\___/ \___/|_|
64+
> ______________________________________________________________________
65+
>
66+
> Earth System Model Evaluation Tool
67+
>
68+
> A community tool for the evaluation of Earth system models.
69+
>
70+
> https://esmvaltool.org
71+
>
72+
> The Earth System Model Evaluation Tool (ESMValTool) is a community
73+
> diagnostics and performance metrics tool for the evaluation of Earth
74+
> System Models (ESMs) that allows for routine comparison of single or
75+
> multiple models, either against predecessor versions or against
76+
> observations.
77+
>
78+
> Tutorial: https://tutorial.esmvaltool.org
79+
> Documentation: https://docs.esmvaltool.org
80+
> Contact: esmvaltool-dev@listserv.dfn.de
81+
>
82+
> If you find this software useful for your research, please cite it using
83+
> https://doi.org/10.5281/zenodo.3387139 for ESMValCore or
84+
> https://doi.org/10.5281/zenodo.3401363 for ESMValTool or
85+
> any of the reference papers listed at https://esmvaltool.org/references/.
86+
>
87+
> Have fun!
88+
>
89+
> 2026-04-02 12:02:54,973 UTC [3235444] INFO Package versions
90+
> 2026-04-02 12:02:54,973 UTC [3235444] INFO ----------------
91+
> 2026-04-02 12:02:54,973 UTC [3235444] INFO ESMValCore: 2.15.0.dev9+g412767341
92+
> 2026-04-02 12:02:54,973 UTC [3235444] INFO ESMValTool: 2.15.0.dev4+g74d7837f0
93+
> 2026-04-02 12:02:54,973 UTC [3235444] INFO ----------------
94+
> 2026-04-02 12:02:54,974 UTC [3235444] INFO Reading configuration files from:
95+
> /home/b/username/ESMValTool/ESMValCore/esmvalcore/config/configurations/defaults (defaults)
96+
> /home/b/username/.config/esmvaltool (default user configuration directory)
97+
> 2026-04-02 12:02:54,974 UTC [3235444] INFO Writing program log files to:
98+
> /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/main_log.txt
99+
> /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/main_log_debug.txt
100+
> /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/cmor_log.txt
101+
> 2026-04-02 12:02:56,104 UTC [3235444] INFO Starting the Earth System Model Evaluation Tool at time: 2026-04-02 12:02:56 UTC
102+
> 2026-04-02 12:02:56,104 UTC [3235444] INFO ----------------------------------------------------------------------
103+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO RECIPE = /home/b/username/ESMValTool/ESMValTool/esmvaltool/recipes/examples/recipe_python.yml
104+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO RUNDIR = /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run
105+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO WORKDIR = /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/work
106+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO PREPROCDIR = /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/preproc
107+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO PLOTDIR = /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/plots
108+
> 2026-04-02 12:02:56,105 UTC [3235444] INFO ----------------------------------------------------------------------
109+
> 2026-04-02 12:02:56,163 UTC [3235444] INFO Running tasks using at most 256 processes
110+
> 2026-04-02 12:02:56,163 UTC [3235444] INFO If your system hangs during execution, it may not have enough memory for keeping this number of tasks in memory.
111+
> 2026-04-02 12:02:56,163 UTC [3235444] INFO If you experience memory problems, try reducing 'max_parallel_tasks' in your configuration.
112+
> 2026-04-02 12:02:56,808 UTC [3235444] INFO Creating tasks from recipe
113+
> 2026-04-02 12:02:56,808 UTC [3235444] INFO Creating tasks for diagnostic map
114+
> 2026-04-02 12:02:56,808 UTC [3235444] INFO Creating diagnostic task map/script1
115+
> 2026-04-02 12:02:56,810 UTC [3235444] INFO Creating preprocessor task map/tas
116+
> 2026-04-02 12:02:56,810 UTC [3235444] INFO Creating preprocessor 'to_degrees_c' task for variable 'tas'
117+
> 2026-04-02 12:02:56,821 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, historical, r1i1p1f1, mon, gn, v20181214
118+
> 2026-04-02 12:02:56,826 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, mon, v1
119+
> 2026-04-02 12:02:56,827 UTC [3235444] INFO PreprocessingTask map/tas created.
120+
> 2026-04-02 12:02:56,827 UTC [3235444] INFO Creating tasks for diagnostic timeseries
121+
> 2026-04-02 12:02:56,827 UTC [3235444] INFO Creating diagnostic task timeseries/script1
122+
> 2026-04-02 12:02:56,828 UTC [3235444] INFO Creating preprocessor task timeseries/tas_amsterdam
123+
> 2026-04-02 12:02:56,828 UTC [3235444] INFO Creating preprocessor 'annual_mean_amsterdam' task for variable 'tas_amsterdam'
124+
> 2026-04-02 12:02:56,833 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, historical, r1i1p1f1, mon, gn, v20181214
125+
> 2026-04-02 12:02:56,837 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, mon, v1
126+
> 2026-04-02 12:02:56,838 UTC [3235444] INFO PreprocessingTask timeseries/tas_amsterdam created.
127+
> 2026-04-02 12:02:56,838 UTC [3235444] INFO Creating preprocessor task timeseries/tas_global
128+
> 2026-04-02 12:02:56,838 UTC [3235444] INFO Creating preprocessor 'annual_mean_global' task for variable 'tas_global'
129+
> 2026-04-02 12:02:56,845 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP6, BCC-ESM1, historical, r1i1p1f1, mon, gn, v20181214,
130+
supplementaries: areacella, fx, 1pctCO2, fx, v20190613
131+
> 2026-04-02 12:02:56,849 UTC [3235444] INFO Found input files for Dataset: tas, Amon, CMIP5, bcc-csm1-1, historical, r1i1p1, mon, v1, supplementaries:
132+
areacella, fx, r0i0p0, fx
133+
> 2026-04-02 12:02:56,850 UTC [3235444] INFO PreprocessingTask timeseries/tas_global created.
134+
> 2026-04-02 12:02:56,850 UTC [3235444] INFO These tasks will be executed: timeseries/tas_amsterdam, map/tas, map/script1, timeseries/script1, timeseries/tas_global
135+
> 2026-04-02 12:02:56,880 UTC [3235444] INFO Wrote recipe with version numbers and wildcards to:
136+
> file:///scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/recipe_python_filled.yml
137+
> 2026-04-02 12:03:15,435 UTC [3235444] INFO Using Dask distributed scheduler (address: tcp://127.0.0.1:46079, dashboard link: http://127.0.0.1:8787/status)
138+
> 2026-04-02 12:03:15,436 UTC [3235444] INFO Running 5 tasks using 5 processes
139+
> 2026-04-02 12:03:15,519 UTC [3236556] INFO Starting task map/tas in process [3236556]
140+
> 2026-04-02 12:03:15,519 UTC [3236558] INFO Starting task timeseries/tas_amsterdam in process [3236558]
141+
> 2026-04-02 12:03:15,520 UTC [3236560] INFO Starting task timeseries/tas_global in process [3236560]
142+
> 2026-04-02 12:03:15,606 UTC [3235444] INFO Progress: 3 tasks running, 2 tasks waiting for ancestors, 0/5 done
143+
> 2026-04-02 12:03:15,805 UTC [3236558] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E)
144+
> 2026-04-02 12:03:16,177 UTC [3236558] INFO Extracting data for Amsterdam, Noord-Holland, Nederland (52.3730796 °N, 4.8924534 °E)
145+
> 2026-04-02 12:03:16,239 UTC [3236556] INFO Computing and saving data for preprocessing task map/tas
146+
> 2026-04-02 12:03:16,384 UTC [3236558] INFO Generated PreprocessorFile: /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/preproc/
147+
timeseries/tas_amsterdam/MultiModelMean_historical_Amon_tas_1850-2000.nc
148+
> 2026-04-02 12:03:17,250 UTC [3236556] INFO map/tas [########################################] | 100% Completed | 1.01 s
149+
> 2026-04-02 12:03:17,252 UTC [3236560] INFO Computing and saving data for preprocessing task timeseries/tas_global
150+
> 2026-04-02 12:03:17,256 UTC [3236556] INFO Successfully completed task map/tas (priority 1) in 0:00:01.736625
151+
> 2026-04-02 12:03:17,414 UTC [3235444] INFO Progress: 2 tasks running, 2 tasks waiting for ancestors, 1/5 done
152+
> 2026-04-02 12:03:17,442 UTC [3236562] INFO Starting task map/script1 in process [3236562]
153+
> 2026-04-02 12:03:17,447 UTC [3236562] INFO Running command ['/work/bd0854/username/utils/mambaforge/envs/esmvaltool/bin/python3.12',
154+
'/home/b/username/ESMValTool/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py',
155+
'/scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/map/script1/settings.yml']
156+
> 2026-04-02 12:03:17,447 UTC [3236562] INFO Writing output to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/work/map/script1
157+
> 2026-04-02 12:03:17,447 UTC [3236562] INFO Writing plots to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/plots/map/script1
158+
> 2026-04-02 12:03:17,447 UTC [3236562] INFO Writing log to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/map/script1/log.txt
159+
> 2026-04-02 12:03:17,447 UTC [3236562] INFO To re-run this diagnostic script, run:
160+
> cd /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/map/script1; MPLBACKEND="Agg" /work/bd0854/username/utils/mambaforge/envs/
161+
esmvaltool/bin/python3.12 /home/b/username/ESMValTool/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py
162+
/scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/map/script1/settings.yml
163+
> 2026-04-02 12:03:17,514 UTC [3235444] INFO Progress: 3 tasks running, 1 tasks waiting for ancestors, 1/5 done
164+
> 2026-04-02 12:03:18,367 UTC [3236560] INFO timeseries/tas_global [########################################] | 100% Completed | 1.10 s
165+
> 2026-04-02 12:03:18,370 UTC [3236558] INFO Computing and saving data for preprocessing task timeseries/tas_amsterdam
166+
> 2026-04-02 12:03:18,394 UTC [3236560] INFO Successfully completed task timeseries/tas_global (priority 4) in 0:00:02.873452
167+
> 2026-04-02 12:03:18,675 UTC [3235444] INFO Progress: 2 tasks running, 1 tasks waiting for ancestors, 2/5 done
168+
> 2026-04-02 12:03:20,566 UTC [3236558] INFO timeseries/tas_amsterdam [########################################] | 100% Completed | 2.18 s
169+
> 2026-04-02 12:03:20,591 UTC [3236558] INFO Successfully completed task timeseries/tas_amsterdam (priority 3) in 0:00:05.071412
170+
> 2026-04-02 12:03:20,712 UTC [3235444] INFO Progress: 1 tasks running, 1 tasks waiting for ancestors, 3/5 done
171+
> 2026-04-02 12:03:20,730 UTC [3236564] INFO Starting task timeseries/script1 in process [3236564]
172+
> 2026-04-02 12:03:20,757 UTC [3236564] INFO Running command ['/work/bd0854/username/utils/mambaforge/envs/esmvaltool/bin/python3.12',
173+
'/home/b/username/ESMValTool/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py',
174+
'/scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/timeseries/script1/settings.yml']
175+
> 2026-04-02 12:03:20,757 UTC [3236564] INFO Writing output to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/work/timeseries/script1
176+
> 2026-04-02 12:03:20,757 UTC [3236564] INFO Writing plots to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/plots/timeseries/script1
177+
> 2026-04-02 12:03:20,757 UTC [3236564] INFO Writing log to /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/timeseries/script1/log.txt
178+
> 2026-04-02 12:03:20,758 UTC [3236564] INFO To re-run this diagnostic script, run:
179+
> cd /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/timeseries/script1; MPLBACKEND="Agg"
180+
/work/bd0854/username/utils/mambaforge/envs/esmvaltool/bin/python3.12 /home/b/username/ESMValTool/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py
181+
/scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/timeseries/script1/settings.yml
182+
> 2026-04-02 12:03:20,812 UTC [3235444] INFO Progress: 2 tasks running, 0 tasks waiting for ancestors, 3/5 done
183+
> 2026-04-02 12:03:28,198 UTC [3236562] INFO Maximum memory used (estimate): 0.3 GB
184+
> 2026-04-02 12:03:28,199 UTC [3236562] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
185+
> 2026-04-02 12:03:28,274 UTC [3236564] INFO Maximum memory used (estimate): 0.3 GB
186+
> 2026-04-02 12:03:28,275 UTC [3236564] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
187+
> 2026-04-02 12:03:28,618 UTC [3236562] INFO Successfully completed task map/script1 (priority 0) in 0:00:11.175930
188+
> 2026-04-02 12:03:28,820 UTC [3235444] INFO Progress: 1 tasks running, 0 tasks waiting for ancestors, 4/5 done
189+
> 2026-04-02 12:03:29,008 UTC [3236564] INFO Successfully completed task timeseries/script1 (priority 2) in 0:00:08.278011
190+
> 2026-04-02 12:03:29,020 UTC [3235444] INFO Progress: 0 tasks running, 0 tasks waiting for ancestors, 5/5 done
191+
> 2026-04-02 12:03:29,020 UTC [3235444] INFO Successfully completed all tasks.
192+
> 2026-04-02 12:03:29,716 UTC [3235444] INFO Wrote recipe with version numbers and wildcards to:
193+
> file:///scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/run/recipe_python_filled.yml
194+
> 2026-04-02 12:03:30,651 UTC [3235444] INFO Wrote recipe output to:
195+
> file:///scratch/b/username/esmvaltool_output/recipe_python_20260402_120254/index.html
196+
> 2026-04-02 12:03:30,651 UTC [3235444] INFO It looks like you are connected to a remote machine via SSH. To show the output html file, you can try the
197+
following command on your local machine:
198+
> server=136.172.124.6 && port=31415 && ssh -t -L ${port}:localhost:${port} username@${server} /work/bd0854/username/utils/mambaforge/envs/esmvaltool/bin/
199+
python3.12 -m http.server ${port} -d /scratch/b/username/esmvaltool_output/recipe_python_20260402_120254
200+
> Then visit http://localhost:31415 in your browser
201+
> 2026-04-02 12:03:30,651 UTC [3235444] INFO If the port 31415 is already in use, you can replace it with any other free one (e.g., 12789). If you are
202+
connected through a jump host, replace the server IP address 136.172.124.6 with your SSH server name
203+
> 2026-04-02 12:03:30,652 UTC [3235444] INFO Ending the Earth System Model Evaluation Tool at time: 2026-04-02 12:03:30 UTC
204+
> 2026-04-02 12:03:30,652 UTC [3235444] INFO Time for running the recipe was: 0:00:34.547424
205+
> 2026-04-02 12:03:31,701 UTC [3235444] INFO Maximum memory used (estimate): 8.3 GB
206+
> 2026-04-02 12:03:31,708 UTC [3235444] INFO Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
207+
> 2026-04-02 12:03:31,733 UTC [3235444] WARNING Input data is not (fully) CMOR-compliant, see /scratch/b/username/esmvaltool_output/
208+
recipe_python_20260402_120254/run/cmor_log.txt for details
209+
> 2026-04-02 12:03:31,733 UTC [3235444] INFO Run was successful
210+
>
211+
> ```
212+
>
213+
{: .solution}
54214
55-
{% include example_output.txt %}
56-
57-
```
58-
{: .output}
59215
60216
> ## Pro tip: ESMValTool search paths
61217
>
@@ -75,17 +231,18 @@ Let's dissect what's happening here.
75231
>
76232
> After the banner and general information, the output starts with some important locations.
77233
>
78-
> 1. Did ESMValTool use the right config file?
234+
> 1. Did ESMValTool use the right config files?
79235
> 1. What is the path to the example recipe?
80236
> 1. What is the main output folder generated by ESMValTool?
81237
> 1. Can you guess what the different output directories are for?
82238
> 1. ESMValTool creates two log files. What is the difference?
83239
>
84240
> > ## Answers
85241
> >
86-
> > 1. The config file should be the one we edited in the previous episode,
87-
> > something like `/home/<username>/.config/esmvaltool/config-user.yml` or
88-
`~/esmvaltool_tutorial/config-user.yml`.
242+
> > 1. The list of directories involves the directory with all default config files
243+
> > (e.g., `../ESMValCore/esmvalcore/config/configurations/defaults`)
244+
> > and should list the (default) user configuration directory with our edited files
245+
> > from the previous episode (default: `~/.config/esmvaltool`).
89246
> > 1. ESMValTool found the recipe in its installation directory,
90247
>> something like
91248
> > `/home/users/username/mambaforge/envs/esmvaltool/bin/esmvaltool/recipes/examples/`
@@ -98,11 +255,11 @@ Let's dissect what's happening here.
98255
> > folder is made inside the output directory specified in the previous
99256
> > episode: `~/esmvaltool_tutorial/esmvaltool_output`.
100257
> > 1. There should be four output folders:
101-
> > - `plots/`: this is where output figures are stored.
102-
> > - `preproc/`: this is where pre-processed data are stored.
103258
> > - `run/`: this is where esmvaltool stores general information about the
104259
> > run, such as log messages and a copy of the recipe file.
105260
> > - `work/`: this is where output files (not figures) are stored.
261+
> > - `preproc/`: this is where pre-processed data are stored.
262+
> > - `plots/`: this is where output figures are stored.
106263
> > 1. The log files are:
107264
> > - `main_log.txt` is a copy of the command-line output
108265
> > - `main_log_debug.txt` contains more detailed information that may be

0 commit comments

Comments
 (0)