Merge branch 'bugfix/2568-output-fenceposting' of github.com:Parcels-Code/Parcels into bugfix/2568-output-fenceposting

Author: fluidnumericsJoe

.github/ISSUE_TEMPLATE/02_bug.yaml

@@ -17,7 +17,7 @@ body:
   - type: "textarea"
     attributes:
       label: "Code sample"
-      description: "If relevant, please provide a code example where this bug is shown as well as any error message. A [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) is preffered as it makes it much easier for developers to identify the cause of the bug. This also allows them quickly determine whether the problem is with your code or with Parcels itself. If you want support on a specific dataset, please [follow our instructions on how to share dataset metadata](https://docs.parcels-code.org/en/main/development/posting-issues.html)"
+      description: "If relevant, please provide a code example where this bug is shown as well as any error message. A [minimal, reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) is preffered as it makes it much easier for developers to identify the cause of the bug. This also allows them quickly determine whether the problem is with your code or with Parcels itself. If you want support on a specific dataset, please [follow our instructions on how to share representative datasets](https://docs.parcels-code.org/en/main/development/posting-issues.html)"
       value: |
         ```python
         # Paste your code within this block

.github/ci/recipe.yaml

@@ -35,7 +35,7 @@ requirements:
     - netcdf4 >=1.7.2
     - numpy >=2.1.0
     - tqdm >=4.50.0
-    - xarray >=2024.5.0
+    - xarray >=2025.8.0,<2026.4.0 # TODO: remove upper pin when https://github.com/UXARRAY/uxarray/issues/1490 is resolved
     - cf_xarray >=0.8.6
     - xgcm >=0.9.0
     - zarr >=2.15.0,!=2.18.0,<3

.github/workflows/ci.yml

@@ -243,3 +243,17 @@ jobs:
           done
         env:
           PREFIX_API_KEY: ${{ secrets.PREFIX_API_KEY }} # zizmor: ignore[secrets-outside-env]
+
+  zizmor:
+    name: GHA Security Analysis using Zizmor
+    runs-on: ubuntu-latest
+    permissions:
+      security-events: write # Required for upload-sarif (used by zizmor-action) to upload SARIF files.
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Run zizmor
+        uses: zizmorcore/zizmor-action@71321a20a9ded102f6e9ce5718a2fcec2c4f70d8 # v0.5.2

.pre-commit-config.yaml

@@ -13,8 +13,9 @@ repos:
     rev: v1.23.1
     hooks:
       - id: zizmor
+        args: ["--offline"]
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.8
+    rev: v0.15.9
     hooks:
       - id: ruff
         name: ruff lint (.py)

docs/development/posting-issues.md

@@ -20,51 +20,96 @@ Following these templates provides structure and ensures that we have all the ne
 Parcels is designed to work with a large range of input datasets.
 
 When extending support for various input datasets, or trying to debug problems
-that only occur with specific datasets, having the dataset metadata is very valuable.
+that only occur with specific datasets, having access to your dataset (or a
+close representation of it) is very valuable.
 
-This metadata could include information such as:
+This could include information such as:
 
 - the nature of the array variables (e.g., via CF compliant metadata)
 - descriptions about the origin of the dataset, or additional comments
 - the shapes and data types of the arrays
+- the grid topology (coordinates and key variables)
 
 This also allows us to see if your metadata is broken/non-compliant with standards - where we can then suggest fixes for you (and maybe we can tell the data provider!).
 Since version 4 of Parcels we rely much more on metadata to discover information about your input data.
 
-Sharing this metadata often provides enough debugging information to solve your problem, instead of having to share a whole dataset.
+Sharing a compact representation of your dataset often provides enough information to solve your problem, without having to share the full dataset (which may be very large or contain sensitive data).
 
-Sharing dataset metadata is made easy in Parcels.
+Parcels makes this easy by replacing irrelevant array data with zeros and saving the result as a compressed Zarr zip store, which is typically small enough to attach directly to a GitHub issue.
 
 ### Step 1. Users
 
 As a user with access to your dataset, you would do:
 
 ```{code-cell}
-import json
+:tags: [hide-cell]
 
+# Generate an example dataset to zip. The user would use their own.
 import xarray as xr
+from parcels._datasets.structured.generic import datasets
+datasets['ds_2d_left'].to_netcdf("my_dataset.nc")
+```
+
+```{code-cell}
+import os
+
+import xarray as xr
+import zarr
+
+from parcels._datasets.utils import replace_arrays_with_zeros
+
+# load your dataset
+ds = xr.open_dataset("my_dataset.nc")  # or xr.open_zarr(...), etc.
+
+# Replace all data arrays with zeros, keeping coordinate metadata.
+# This keeps array shapes and metadata while removing actual data.
+#
+# You can customise `except_for` to also retain actual values for specific variables:
+#   except_for='coords'         — keep coordinate arrays (useful for grid topology)
+#   except_for=['lon', 'lat']   — keep a specific list of variables
+#   except_for=None   — remove all arrays (useful to know about dtypes, structure, and metadata). This is the default for the function.
+ds_trimmed = replace_arrays_with_zeros(ds, except_for = None)
 
-# defining an example dataset to illustrate
-# (you would use `xr.open_dataset(...)` instead)
-ds = xr.Dataset(attrs={"description": "my dataset"})
+# Save to a zipped Zarr store - replace `my_dataset` with a more informative name
+with zarr.storage.ZipStore("my_dataset.zip", mode='w') as store:
+    ds_trimmed.to_zarr(store)
 
-output_file = "my_dataset.json"
-with open(output_file, "w") as f:
-    json.dump(ds.to_dict(data=False), f)  # write your dataset to a JSON excluding array data
+size_mb_original = os.path.getsize("my_dataset.nc") / 1e6
+print(f"Original size: {size_mb_original:.1f} MB")
+
+# Check the file size (aim for < 25 MB so it can be attached to a GitHub issue)
+size_mb = os.path.getsize("my_dataset.zip") / 1e6
+print(f"Zip store size: {size_mb:.1f} MB")
 ```
 
-Then attach the JSON file written above alongside your issue
+Then attach the zip file written above alongside your issue.
+
+If the file is larger than 25 MB, try passing `except_for=None` (the default)
+to ensure all arrays are zeroed out. If it is still too large, consider
+subsetting your dataset to a smaller spatial or temporal region before saving.
 
 ### Step 2. Maintainers and developers
 
-As developers looking to inspect the metadata, we would do:
+As developers looking to inspect the dataset, we would do:
+
+```{code-cell}
+import xarray as xr
+import zarr
+
+ds = xr.open_zarr(zarr.storage.ZipStore("my_dataset.zip", mode="r"))
+ds
+```
 
 ```{code-cell}
-from parcels._datasets.utils import from_xarray_dataset_dict
+:tags: [hide-cell]
+
+# Cleanup files in doc build process
+del ds
+from pathlib import Path
+Path("my_dataset.zip").unlink()
+Path("my_dataset.nc").unlink()
 
-with open(output_file) as f:
-    d = json.load(f)
-ds = from_xarray_dataset_dict(d)
 ```
 
-From there we can take a look the metadata of your dataset!
+From there we can take a look at the structure, metadata, and grid topology of your dataset!
+This also makes it straightforward for us to add this dataset to our test suite.

docs/getting_started/tutorial_output.ipynb

@@ -35,7 +35,8 @@
     "import numpy as np\n",
     "import xarray as xr\n",
     "\n",
-    "import parcels"
+    "import parcels\n",
+    "import parcels.tutorial"
    ]
   },
   {
@@ -52,11 +53,9 @@
    "outputs": [],
    "source": [
     "# Load the CopernicusMarine data in the Agulhas region from the example_datasets\n",
-    "example_dataset_folder = parcels.download_example_dataset(\n",
-    "    \"CopernicusMarine_data_for_Argo_tutorial\"\n",
+    "ds_fields = parcels.tutorial.open_dataset(\n",
+    "    \"CopernicusMarine_data_for_Argo_tutorial/data\"\n",
     ")\n",
-    "\n",
-    "ds_fields = xr.open_mfdataset(f\"{example_dataset_folder}/*.nc\", combine=\"by_coords\")\n",
     "ds_fields.load()  # load the dataset into memory\n",
     "\n",
     "# Convert to SGRID-compliant dataset and create FieldSet\n",
@@ -558,7 +557,7 @@
  "metadata": {
   "celltoolbar": "Metagegevens bewerken",
   "kernelspec": {
-   "display_name": "test-notebooks",
+   "display_name": "default",
    "language": "python",
    "name": "python3"
   },
@@ -572,7 +571,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.14.2"
+   "version": "3.14.3"
   }
  },
  "nbformat": 4,

docs/getting_started/tutorial_quickstart.md

@@ -20,6 +20,7 @@ and writing output files that can be read with xarray.
 import numpy as np
 import xarray as xr
 import parcels
+import parcels.tutorial
 ```
 
 ## Input flow fields: `FieldSet`
@@ -29,11 +30,8 @@ hydrodynamics fields in which the particles are tracked. Here we provide an exam
 [Global Ocean Physics Reanalysis](https://doi.org/10.48670/moi-00021) from the Copernicus Marine Service.
 
 ```{code-cell}
-example_dataset_folder = parcels.download_example_dataset(
-    "CopernicusMarine_data_for_Argo_tutorial"
-)
+ds_fields = parcels.tutorial.open_dataset("CopernicusMarine_data_for_Argo_tutorial/data")
 
-ds_fields = xr.open_mfdataset(f"{example_dataset_folder}/*.nc", combine="by_coords")
 ds_fields.load()  # load the dataset into memory
 ds_fields
 ```

docs/user_guide/examples/explanation_kernelloop.md

@@ -53,13 +53,10 @@ import numpy as np
 import xarray as xr
 
 import parcels
+import parcels.tutorial
 
 # Load the CopernicusMarine data in the Agulhas region from the example_datasets
-example_dataset_folder = parcels.download_example_dataset(
-    "CopernicusMarine_data_for_Argo_tutorial"
-)
-
-ds_fields = xr.open_mfdataset(f"{example_dataset_folder}/*.nc", combine="by_coords")
+ds_fields = parcels.tutorial.open_dataset("CopernicusMarine_data_for_Argo_tutorial/data")
 ds_fields.load()  # load the dataset into memory
 
 # Create an idealised wind field and add it to the dataset

docs/user_guide/examples/tutorial_Argofloats.ipynb

@@ -110,14 +110,13 @@
     "import xarray as xr\n",
     "\n",
     "import parcels\n",
+    "import parcels.tutorial\n",
     "\n",
     "# Load the CopernicusMarine data in the Agulhas region from the example_datasets\n",
-    "example_dataset_folder = parcels.download_example_dataset(\n",
-    "    \"CopernicusMarine_data_for_Argo_tutorial\"\n",
+    "ds_fields = parcels.tutorial.open_dataset(\n",
+    "    \"CopernicusMarine_data_for_Argo_tutorial/data\"\n",
     ")\n",
     "\n",
-    "ds_fields = xr.open_mfdataset(f\"{example_dataset_folder}/*.nc\", combine=\"by_coords\")\n",
-    "\n",
     "# TODO check how we can get good performance without loading full dataset in memory\n",
     "ds_fields.load()  # load the dataset into memory\n",
     "\n",

docs/user_guide/examples/tutorial_croco_3D.ipynb

@@ -34,16 +34,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import matplotlib.pyplot as plt\n",
-    "import numpy as np\n",
-    "import xarray as xr\n",
-    "\n",
-    "import parcels\n",
-    "\n",
-    "data_folder = parcels.download_example_dataset(\"CROCOidealized_data\")\n",
-    "ds_fields = xr.open_dataset(data_folder / \"CROCO_idealized.nc\")\n",
-    "\n",
-    "ds_fields.load();  # Preload data to speed up access"
+    "import matplotlib.pyplot as plt\nimport numpy as np\nimport xarray as xr\n\nimport parcels\nimport parcels.tutorial\n\nds_fields = parcels.tutorial.open_dataset(\"CROCOidealized_data/data\")\n\nds_fields.load();  # Preload data to speed up access"
    ]
   },
   {