Many rechunked Zarr stores contain metadata but no data (NaN-filled arrays)

[cadejs]

A significant fraction of the rechunked_data/ Zarr stores on OSN contain correct metadata
(coordinates, dimensions, attributes) but NaN-filled data arrays. This primarily affects
temperature variables (tasmax, tasmin) across multiple methods.

  import numpy as np
  import xarray as xr                                                                               
   
  DATASETS = [                                                                                      
      # Temperature -- return NaN                           
      "https://rice1.osn.mghpcc.org/carbonplan/cp-cmip/version1/rechunked_data/GARD-SV/CMIP.CCCma.Ca
  nESM5.historical.r1i1p1f1.day.GARD-SV.tasmax.zarr",                                               
      "https://rice1.osn.mghpcc.org/carbonplan/cp-cmip/version1/rechunked_data/GARD-SV/CMIP.CCCma.Ca
  nESM5.historical.r1i1p1f1.day.GARD-SV.tasmin.zarr",                                               
      "https://rice1.osn.mghpcc.org/carbonplan/cp-cmip/version1/rechunked_data/DeepSD/CMIP.CCCma.Can
  ESM5.historical.r1i1p1f1.day.DeepSD.tasmax.zarr",                                                 
      # Precipitation -- return valid data                  
      "https://rice1.osn.mghpcc.org/carbonplan/cp-cmip/version1/rechunked_data/GARD-SV/CMIP.CCCma.Ca
  nESM5.historical.r1i1p1f1.day.GARD-SV.pr.zarr",                                                   
  ]                                                                                                 
                                                                                                    
  for url in DATASETS:                                      
      ds = xr.open_zarr(url, chunks={})
      var = list(ds.data_vars)[0]                                                                   
      val = float(ds[var].isel(time=0, lat=len(ds.lat)//2, lon=len(ds.lon)//2).values)              
      status = "NaN" if np.isnan(val) else f"{val:.2f}"                                             
      print(f"{var:>6}  {status:>8}  {dict(ds.sizes)}  {url.split('rechunked_data/')[1]}")          
      ds.close()                                                                                    
                                       

Output

tasmax NaN {'time': 23741, 'lat': 721, 'lon': 1440}
GARD-SV/CMIP.CCCma.CanESM5.historical...tasmax.zarr
tasmin NaN {'time': 23741, 'lat': 721, 'lon': 1440}
GARD-SV/CMIP.CCCma.CanESM5.historical...tasmin.zarr
tasmax NaN {'time': 23741, 'lat': 720, 'lon': 1440}
DeepSD/CMIP.CCCma.CanESM5.historical...tasmax.zarr
pr 55.12 {'time': 23741, 'lat': 721, 'lon': 1440}
GARD-SV/CMIP.CCCma.CanESM5.historical...pr.zarr

Scope

We audited all 7286 Zarr files across the 5 daily methods (GARD-SV, GARD-MV, DeepSD, DeepSD-BC,
MACA). Approximate valid rates:

┌──────────┬───────┐
│ Variable │ Valid │
├──────────┼───────┤
│ pr │ ~82% │
├──────────┼───────┤
│ tasmax │ ~25% │
├──────────┼───────┤
│ tasmin │ ~22% │
└──────────┴───────┘

The pattern is not strictly by variable, e.g. DeepSD/CanESM5/ssp245/tasmin has valid data while
DeepSD/CanESM5/historical/tasmax does not.

Is there another way to download the datasets or are they stored elsewhere?

[norlandrhagen]

Hey there @cadejs, thanks for the report. We have another copy in the same OSN pod at:https://rice1.osn.mghpcc.org/carbonplan/cp-cmip/version1/data/. Not sure if we're also missing data there as well. If so, there should be access on MSFT planetary computer via Azure.

[cadejs]

I am double checking at the moment, but it appears to also have some missing data on the OSN pod. I can update with more specifics if that is helpful.

What is the data listed under on the MSFT Planetary Computer?

[norlandrhagen]

@cadejs if you don't mind that would be super helpful! It would be nice to know if the data/ vs rechunked_data/ have the same missing bits.

Here is the dataset.md file that had the Azure store links.

[cadejs]

@norlandrhagen I will get that to you once I finish running the code that checks both!

In the meantime it looks like the blob is not set to allow for public access. Is there a process by which to gain access?
This XML file does not appear to have any style information associated with it. The document tree is shown below. <Error> <Code>PublicAccessNotPermitted</Code> <Message>Public access is not permitted on this storage account. RequestId: Time:2026-03-13T14:43:05.0177929Z</Message> </Error>

[norlandrhagen]

Thanks! That'll be super helpful.

Ah, that's a bummer. It was public, but maybe MSFT planetary computer changed their bucket access policy? FWIW, we don't have admin access on the MSFT bucket.

[maxrjones]

FYI you can use obstore's credential provider to authenticate to the data on MS OpenPC:

# /// script
# dependencies = [
#   "xarray @ git+https://github.com/pydata/xarray@main",
#   "zarr @ git+https://github.com/zarr-developers/zarr-python@main",
#   "numpy",
#   "obstore",
#   "requests",
#   "dask",
# ]
# ///

from obstore.store import AzureStore
from obstore.auth.planetary_computer import PlanetaryComputerCredentialProvider
import xarray as xr
import numpy as np
from zarr.storage import ObjectStore
url = "https://cpdataeuwest.blob.core.windows.net/cp-cmip/version1/data/GARD-SV/CMIP.CCCma.CanESM5.historical.r1i1p1f1.day.GARD-SV.pr.zarr"
# Construct an AzureStore with this credential provider.
#
# The account, container, and container prefix are passed down to AzureStore
# automatically.
store = AzureStore(credential_provider=PlanetaryComputerCredentialProvider(url))
zs = ObjectStore(store)

ds = xr.open_zarr(zs, chunks={})
var = list(ds.data_vars)[0]                                                                   
val = float(ds[var].isel(time=0, lat=len(ds.lat)//2, lon=len(ds.lon)//2).values)              
status = "NaN" if np.isnan(val) else f"{val:.2f}"                                             
print(f"{var:>6}  {status:>8}  {dict(ds.sizes)}  {url.split('data/')[1]}")          
ds.close()                                                                                    

I was looking at this in order to review zarr-developers/zarr-python#3748, since the cmip6-downscaling examples could really benefit from that feature. IIRC the original data should've been written with write_empty_chunks=True.