08 · Stay lazy until the answer is requested¶

Context¶

Real cube archives quickly outgrow memory. Lazy arrays let us describe a method first and materialize only the final analysis.

Question¶

Can a composed pipe preserve Dask-backed execution while computing a compact spatial summary from real observations?

Analysis story¶

We open the reviewed fixture with chunks, compose anomaly and variance verbs, confirm that the result stays lazy, and compute only the final map.

Data used in this lesson¶

Every value comes from the PRISM Group at Oregon State University's AN91d daily 4 km climate product. This repository carries a small Boulder-region extract for 1–30 January 2024 so the lesson runs offline without replacing observations with generated values. The data validation page records source URLs, terms, checksums, bounds, units, and acceptance tests.

Prepare · Open the official extract as a chunked cube¶

In [1]:

Copied!





from pathlib import Path
import xarray as xr

# The checked-in extract makes the lesson reproducible without a live service.
data_path = next(
    candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc"
    for candidate in (Path.cwd(), *Path.cwd().parents)
    if (candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc").exists()
)
cube = xr.open_dataset(
    data_path,
    engine="scipy",
    chunks={"time": 10, "y": 12, "x": 12},
)["tmax"]

assert cube.attrs["is_synthetic"] == 0
assert hasattr(cube.data, "chunks")
cube
from pathlib import Path
import xarray as xr

# The checked-in extract makes the lesson reproducible without a live service.
data_path = next(
    candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc"
    for candidate in (Path.cwd(), *Path.cwd().parents)
    if (candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc").exists()
)
cube = xr.open_dataset(
    data_path,
    engine="scipy",
    chunks={"time": 10, "y": 12, "x": 12},
)["tmax"]

assert cube.attrs["is_synthetic"] == 0
assert hasattr(cube.data, "chunks")
cube

Out[1]:

<xarray.DataArray 'tmax' (time: 30, y: 24, x: 24)> Size: 69kB
dask.array<open_dataset-tmax, shape=(30, 24, 24), dtype=float32, chunksize=(10, 12, 12), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 240B 2024-01-01 2024-01-02 ... 2024-01-30
  * y        (y) float64 192B 40.5 40.46 40.42 40.37 ... 39.67 39.62 39.58 39.54
  * x        (x) float64 192B -105.7 -105.7 -105.7 ... -104.9 -104.8 -104.8
Attributes:
    long_name:        daily maximum air temperature
    standard_name:    air_temperature
    units:            degC
    source_variable:  PRISM tmax
    is_synthetic:     0

xarray.DataArray

'tmax'

time: 30
y: 24
x: 24

dask.array<chunksize=(10, 12, 12), meta=np.ndarray>

	Array	Chunk
Bytes	67.50 kiB	5.62 kiB
Shape	(30, 24, 24)	(10, 12, 12)
Dask graph	12 chunks in 2 graph layers
Data type	float32 numpy.ndarray

Coordinates: (3)

time

(time)

datetime64[ns]

2024-01-01 ... 2024-01-30

array(['2024-01-01T00:00:00.000000000', '2024-01-02T00:00:00.000000000',
       '2024-01-03T00:00:00.000000000', '2024-01-04T00:00:00.000000000',
       '2024-01-05T00:00:00.000000000', '2024-01-06T00:00:00.000000000',
       '2024-01-07T00:00:00.000000000', '2024-01-08T00:00:00.000000000',
       '2024-01-09T00:00:00.000000000', '2024-01-10T00:00:00.000000000',
       '2024-01-11T00:00:00.000000000', '2024-01-12T00:00:00.000000000',
       '2024-01-13T00:00:00.000000000', '2024-01-14T00:00:00.000000000',
       '2024-01-15T00:00:00.000000000', '2024-01-16T00:00:00.000000000',
       '2024-01-17T00:00:00.000000000', '2024-01-18T00:00:00.000000000',
       '2024-01-19T00:00:00.000000000', '2024-01-20T00:00:00.000000000',
       '2024-01-21T00:00:00.000000000', '2024-01-22T00:00:00.000000000',
       '2024-01-23T00:00:00.000000000', '2024-01-24T00:00:00.000000000',
       '2024-01-25T00:00:00.000000000', '2024-01-26T00:00:00.000000000',
       '2024-01-27T00:00:00.000000000', '2024-01-28T00:00:00.000000000',
       '2024-01-29T00:00:00.000000000', '2024-01-30T00:00:00.000000000'],
      dtype='datetime64[ns]')

y

(y)

float64

40.5 40.46 40.42 ... 39.58 39.54

standard_name :: latitude
units :: degrees_north

array([40.5     , 40.458333, 40.416667, 40.375   , 40.333333, 40.291667,
       40.25    , 40.208333, 40.166667, 40.125   , 40.083333, 40.041667,
       40.      , 39.958333, 39.916667, 39.875   , 39.833333, 39.791667,
       39.75    , 39.708333, 39.666667, 39.625   , 39.583333, 39.541667])

x

(x)

float64

-105.7 -105.7 ... -104.8 -104.8

standard_name :: longitude
units :: degrees_east

array([-105.75    , -105.708333, -105.666667, -105.625   , -105.583333,
       -105.541667, -105.5     , -105.458333, -105.416667, -105.375   ,
       -105.333333, -105.291667, -105.25    , -105.208333, -105.166667,
       -105.125   , -105.083333, -105.041667, -105.      , -104.958333,
       -104.916667, -104.875   , -104.833333, -104.791667])

Attributes: (5)
long_name :
daily maximum air temperature
standard_name :
air_temperature
units :
degC
source_variable :
PRISM tmax
is_synthetic :
0

Pipe · Describe the method without triggering computation¶

In [2]:

Copied!





from cubedynamics import pipe, verbs as v

lazy_variability = (
    pipe(cube)
    | v.anomaly(dim="time")
    | v.variance(dim="time")
).unwrap()

assert hasattr(lazy_variability.data, "chunks")
lazy_variability
from cubedynamics import pipe, verbs as v

lazy_variability = (
    pipe(cube)
    | v.anomaly(dim="time")
    | v.variance(dim="time")
).unwrap()

assert hasattr(lazy_variability.data, "chunks")
lazy_variability

Out[2]:

<xarray.DataArray 'tmax' (time: 1, y: 24, x: 24)> Size: 2kB
dask.array<getitem, shape=(1, 24, 24), dtype=float32, chunksize=(1, 12, 12), chunktype=numpy.ndarray>
Coordinates:
  * y        (y) float64 192B 40.5 40.46 40.42 40.37 ... 39.67 39.62 39.58 39.54
  * x        (x) float64 192B -105.7 -105.7 -105.7 ... -104.9 -104.8 -104.8
Dimensions without coordinates: time
Attributes:
    long_name:        daily maximum air temperature
    standard_name:    air_temperature
    units:            degC
    source_variable:  PRISM tmax
    is_synthetic:     0

Figure · Compute only the final spatial answer¶

In [3]:

Copied!





import matplotlib.pyplot as plt

variability = lazy_variability.compute()
assert not hasattr(variability.data, "chunks")

fig, ax = plt.subplots(figsize=(6.5, 4.5), constrained_layout=True)
variability.plot(ax=ax, cmap="viridis", cbar_kwargs={"label": "Anomaly variance (°C²)"})
ax.set_title("January maximum-temperature variability")
plt.show()
import matplotlib.pyplot as plt

variability = lazy_variability.compute()
assert not hasattr(variability.data, "chunks")

fig, ax = plt.subplots(figsize=(6.5, 4.5), constrained_layout=True)
variability.plot(ax=ax, cmap="viridis", cbar_kwargs={"label": "Anomaly variance (°C²)"})
ax.set_title("January maximum-temperature variability")
plt.show()

No description has been provided for this image

What the figure tells us¶

The final map identifies locations with more variable departures while the intermediate anomaly cube remained lazy.

Try the next variation¶

Change chunk sizes and confirm that final values remain identical.

Data used¶

Field	Frozen analysis input
Provider	PRISM Group, Oregon State University
Product	AN91d daily 4 km time series
Dates	2024-01-01 to 2024-01-30
Fixture	`tests/fixtures/real_data/prism_boulder_january_2024.nc`
Provenance record	`tests/fixtures/real_data/prism_boulder_january_2024.provenance.json`

The PRISM source reference describes current catalog support; the fixture record above identifies the observations used here. Data validation documents checksums and acceptance checks. The analytical baseline and thresholds belong to this story, not the provider.

Reproduce¶

Clone the repository, then run these commands from its root:

python -m pip install -e ".[vignettes]"
python scripts/run_vignettes.py docs/vignettes/lazy_composition.ipynb

No network is needed after installation. Open the downloaded notebook in Jupyter and run all cells to see the same figures. The website executes these cells during its strict build. Environment setup and the vignette contract explain the workflow.