@@ -11,7 +11,6 @@ objectives:
1111- Understand how ESMValTool is configured
1212- Prepare a personalized ESMValTool configuration
1313- Configure ESMValTool to use stored climate data and to download climate data
14-
1514keypoints :
1615- ESMValTool can be configured through YAML files located in `~/.config/esmvaltool` or command line arguments
1716- The final configuration is created by merging the contents of all YAML files and command line arguments
@@ -60,7 +59,7 @@ and then modify it if needed:
6059 nano ~ /.config/esmvaltool/config-user.yml
6160~~~
6261
63- If `` nano `` does not work on your system, or if you prefer a different editor,
62+ If `` nano `` does not work on your system, or if you prefer a different editor,
6463any other editor can be used, e.g. `` vim `` .
6564
6665This file contains the information for:
@@ -73,27 +72,27 @@ This file contains the information for:
7372
7473> ## Text editor side note
7574>
76- > No matter what editor you use, you will need to know where it searches
77- > for and saves files. If you start it from the shell, it will (probably)
78- > use your current working directory as its default location. We use `` nano ``
79- > in examples here because it is one of the least complex text editors.
75+ > No matter what editor you use, you will need to know where it searches for
76+ > and saves files. If you start it from the shell, it will (probably) use your
77+ > current working directory as its default location. We use `` nano `` in examples
78+ > here because it is one of the least complex text editors.
8079> Press <kbd >ctrl</kbd > + <kbd >O</kbd > to save the file,
8180> and then <kbd >ctrl</kbd > + <kbd >X</kbd > to exit `` nano `` .
8281 {: .callout}
8382
8483## Destination directory
8584
86- The example configuration file contains the option `` output_dir `` , which is
87- the rootpath where ESMValTool will store its output folders containing
88- e.g. figures, data, logs, etc. With every run, ESMValTool automatically
89- generates a new output folder determined by recipe name, and date and time
90- using the format: YYYYMMDD_HHMMSS.
85+ The example configuration file contains the option `` output_dir `` , which is the
86+ rootpath where ESMValTool will store its output folders containing e.g. figures,
87+ data, logs, etc. With every run, ESMValTool automatically generates a new output
88+ folder determined by recipe name, and date and time using the format:
89+ YYYYMMDD_HHMMSS.
9190
9291> ## Set the destination directory
9392>
94- > Let's name our destination directory `` esmvaltool_output `` in the current directory.
95- > ESMValTool should write the output to this path, so make sure you have the disk space
96- > to write output to this directory.
93+ > Let's name our destination directory `` esmvaltool_output `` in the current
94+ > directory. ESMValTool should write the output to this path, so make sure you
95+ > have the disk space to write output to this directory.
9796> How do we set this in the ` config-user.yml ` ?
9897>
9998>> ## Solution
@@ -126,12 +125,12 @@ Most of these settings are fairly self-explanatory.
126125>> # # Solution
127126>>
128127>> If the option ``remove_preproc_dir`` is set to ``false``, then the
129- >> ``preproc/`` directory contains all the pre-processed data and the
130- >> metadata interface files.
131- >> If the option ``save_intermediary_cubes`` is set to ``true``
132- >> then data will also be saved after each preprocessor step in the folder
133- >> ``preproc``. Note that saving all intermediate results to file will result
134- >> in a considerable slowdown, and can quickly fill your disk.
128+ >> ``preproc/`` directory contains all the pre-processed data and the metadata
129+ >> interface files.
130+ >> If the option ``save_intermediary_cubes`` is set to ``true`` then data will
131+ >> also be saved after each preprocessor step in the folder ``preproc``. Note
132+ >> that saving all intermediate results to file will result in a considerable
133+ >> slowdown, and can quickly fill your disk.
135134> {: .solution}
136135{: .challenge}
137136
@@ -141,36 +140,36 @@ Most of these settings are fairly self-explanatory.
141140> # # Auxiliary data directory
142141>
143142> The ``auxiliary_data_dir`` setting is the path where any required additional
144- auxiliary data files are stored. This location allows us to tell the diagnostic
145- script where to find the files if they can not be downloaded at runtime. This
146- option should not be used for model or observational datasets, but for data
147- files (e.g. shape files) used in plotting such as coastline descriptions and
148- if you want to feed some additional data (e.g. shape files) to your recipe.
143+ > auxiliary data files are stored. This location allows us to tell the diagnostic
144+ > script where to find the files if they can not be downloaded at runtime. This
145+ > option should not be used for model or observational datasets, but for data
146+ > files (e.g. shape files) used in plotting such as coastline descriptions and
147+ > if you want to feed some additional data (e.g. shape files) to your recipe.
149148>
150149>```yaml
151150> auxiliary_data_dir: ~/auxiliary_data
152151> ```
153152> See more information in ESMValTool
154- [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/
153+ > [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/
155154> quickstart/configure.html?highlight=auxiliary_data#top-level-configuration-options).
156155{: .callout}
157156
158157> # # Number of parallel tasks
159158>
160159> This option enables you to perform parallel processing. You can choose the
161- number of tasks in parallel as 1/2/3/4/... or you can set it to ``null``. That
162- tells ESMValTool to use the maximum number of available CPUs. For the purpose of
163- the tutorial, please set ESMValTool use only 1 cpu :
160+ > number of tasks in parallel as 1/2/3/4/... or you can set it to ``null``. That
161+ > tells ESMValTool to use the maximum number of available CPUs. For the purpose
162+ > of the tutorial, please set ESMValTool use only 1 cpu:
164163>
165164>```yaml
166165> max_parallel_tasks: 1
167166> ```
168167>
169168> In general, if you run out of memory, try setting ``max_parallel_tasks`` to 1.
170- Then, check the amount of memory you need for that by inspecting the file
171- ` ` run/resource_usage.txt`` in the output directory. Using the number there you
172- can increase the number of parallel tasks again to a reasonable number for the
173- amount of memory available in your system.
169+ > Then, check the amount of memory you need for that by inspecting the file
170+ > ``run/resource_usage.txt`` in the output directory. Using the number there you
171+ > can increase the number of parallel tasks again to a reasonable number for the
172+ > amount of memory available in your system.
174173{: .callout}
175174
176175
@@ -183,12 +182,12 @@ In addition another custom configuration directory can be specified via the
183182We will learn how to do this in the
184183[next lesson]({{ page.root }}{% link _episodes/04-recipe.md %}).
185184
186- It is possible to have several configuration files with different purposes,
187- for example : ` ` dask_options.yml``, ``data_sources.yml``.
188- In this case, ESMValTool searches for all YAML files within each of the
189- configuration directories and merges them together. How this is done is
190- explained [here](https://docs.esmvaltool.org/projects/ESMValCore/en/
191- latest/quickstart/ configure.html#yaml-files).
185+ It is possible to have several configuration files with different purposes, for
186+ example : ` ` dask_options.yml``, ``data_sources.yml``.
187+ In this case, ESMValTool searches for all YAML files within each of the
188+ configuration directories and merges them together. How this is done is explained
189+ [here](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart /
190+ configure.html#yaml-files).
192191
193192To show the final configuration that is actually used when running ESMValTool, you can use
194193
@@ -204,7 +203,7 @@ for input data based on their source (e.g.
204203CMIP6, CMIP5, obs4mips, OBS6, OBS). For example, CMIP is used for a dataset from
205204the Climate Model Intercomparison Project whereas OBS may be
206205used for an observational dataset.
207- More information about the projects used in ESMValTool is available in the
206+ More information about the projects used in ESMValTool is available in the
208207[documentation](https://docs.esmvaltool.org/projects/esmvalcore/en/latest/
209208quickstart/find_data.html). The ``data`` section for each project in the configuration
210209files defines sources of input data. The easiest way to get started with these is to
@@ -245,12 +244,11 @@ copy the appropriate configuration file by running
245244 esmvaltool config copy data-intake-esgf.yml
246245~~~
247246
248- Additionally, it is necessary to configure [intake-esgf]
249- (https://intake-esgf.readthedocs.io/en/stable/configure.html).
250- This can be done by updating the `local_cache` and `esg_dataroot`
251- with your desired download directory in your
252- intake-esgf configuration file located at `~/.config/intake-esgf/conf.yaml`.
253- The updated file should look like this :
247+ Additionally, it is necessary to configure
248+ [intake-esgf](https://intake-esgf.readthedocs.io/en/stable/configure.html).
249+ This can be done by updating the `local_cache` and `esg_dataroot` with your
250+ desired download directory in your intake-esgf configuration file located at
251+ `~/.config/intake-esgf/conf.yaml`. The updated file should look like this :
254252> # # conf.yml
255253>
256254> ```yaml
@@ -331,7 +329,7 @@ The updated file should look like this:
331329>>```
332330>>
333331>> - Are you working on your local machine and you want to download missing data using ESMValTool?
334- >> You need to configure `intake-esgf` (see above) ans add the root path of the folder where the data
332+ >> You need to configure `intake-esgf` (see above) ans add the root path of the folder where the data
335333>> has been downloaded to in `data-local-esmvaltool.yml` as specified in the `esgf-cache`.
336334>>
337335>> ```yaml
@@ -373,17 +371,20 @@ The updated file should look like this:
373371>> esmvaltool config copy data-hpc-dkrz.yml
374372>>```
375373>>
376- >> - For more information about configure the data sources, see also the ESMValTool
377- >> [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/
378- configure.html#project-specific-configuration).
374+ >> - For more information about configure the data sources, see also the
375+ >> [ESMValTool documentation](
376+ >> https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/
377+ >> configure.html#project-specific-configuration).
379378> {: .solution}
380379{: .challenge}
381380
382381> # # Configuration via command line
383382>
384- > In addition, all configuration options can also be specified via the command line and those settings will
385- > overwrite any setting given by the YAML files. You can find more information in the
386- > [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#command-line-arguments).
383+ > In addition, all configuration options can also be specified via the command
384+ > line and those settings will overwrite any setting given by the YAML files. You
385+ > can find more information in the
386+ > [documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/
387+ > quickstart/configure.html#command-line-arguments).
387388>
388389{: .callout}
389390
0 commit comments