08 · Stay lazy until the answer is requested¶

Context¶

Real cube archives quickly outgrow memory. Lazy arrays let us describe a method first and materialize only the final analysis.

Question¶

Can a composed pipe preserve Dask-backed execution while computing a compact spatial summary from real observations?

Analysis story¶

We open the reviewed fixture with chunks, compose anomaly and variance verbs, confirm that the result stays lazy, and compute only the final map.

Data used in this lesson¶

Every value comes from the PRISM Group at Oregon State University's AN91d daily 4 km climate product. This repository carries a small Boulder-region extract for 1–30 January 2024 so the lesson runs offline without replacing observations with generated values. The data validation page records source URLs, terms, checksums, bounds, units, and acceptance tests.

In [1]:

runtime-identity

# Record the exact code imported by this notebook kernel.
import cubedynamics as cd

print(cd.version_info())

# Record the exact code imported by this notebook kernel. import cubedynamics as cd print(cd.version_info())

CubeDynamics 0.1.0rc3
Artifact: development checkout
Code: git:a852137944cabf238e9321f2eecf09e483c9b9f5
Imported from: /home/runner/work/cubedynamics/cubedynamics/src/cubedynamics
Distribution: /opt/hostedtoolcache/Python/3.11.16/x64/lib/python3.11/site-packages

Prepare · Open the official extract as a chunked cube¶

In [2]:

from pathlib import Path
import xarray as xr

# The checked-in extract makes the lesson reproducible without a live service.
data_path = next(
    candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc"
    for candidate in (Path.cwd(), *Path.cwd().parents)
    if (candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc").exists()
)
cube = xr.open_dataset(
    data_path,
    engine="scipy",
    chunks={"time": 10, "y": 12, "x": 12},
)["tmax"]

assert cube.attrs["is_synthetic"] == 0
assert hasattr(cube.data, "chunks")
cube

from pathlib import Path import xarray as xr # The checked-in extract makes the lesson reproducible without a live service. data_path = next( candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc" for candidate in (Path.cwd(), *Path.cwd().parents) if (candidate / "tests" / "fixtures" / "real_data" / "prism_boulder_january_2024.nc").exists() ) cube = xr.open_dataset( data_path, engine="scipy", chunks={"time": 10, "y": 12, "x": 12}, )["tmax"] assert cube.attrs["is_synthetic"] == 0 assert hasattr(cube.data, "chunks") cube

Out[2]:

<xarray.DataArray 'tmax' (time: 30, y: 24, x: 24)> Size: 69kB
dask.array<open_dataset-tmax, shape=(30, 24, 24), dtype=float32, chunksize=(10, 12, 12), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 240B 2024-01-01 2024-01-02 ... 2024-01-30
  * y        (y) float64 192B 40.5 40.46 40.42 40.37 ... 39.67 39.62 39.58 39.54
  * x        (x) float64 192B -105.7 -105.7 -105.7 ... -104.9 -104.8 -104.8
Attributes:
    long_name:        daily maximum air temperature
    standard_name:    air_temperature
    units:            degC
    source_variable:  PRISM tmax
    is_synthetic:     0

xarray.DataArray

'tmax'

• time: 30
• y: 24
• x: 24

• dask.array<chunksize=(10, 12, 12), meta=np.ndarray>

  Array Chunk Bytes 67.50 kiB 5.62 kiB Shape (30, 24, 24) (10, 12, 12) Dask graph 12 chunks in 2 graph layers Data type float32 numpy.ndarray

• Coordinates: (3)
  • time
    (time)
    datetime64[ns]
    2024-01-01 ... 2024-01-30

    array(['2024-01-01T00:00:00.000000000', '2024-01-02T00:00:00.000000000',
           '2024-01-03T00:00:00.000000000', '2024-01-04T00:00:00.000000000',
           '2024-01-05T00:00:00.000000000', '2024-01-06T00:00:00.000000000',
           '2024-01-07T00:00:00.000000000', '2024-01-08T00:00:00.000000000',
           '2024-01-09T00:00:00.000000000', '2024-01-10T00:00:00.000000000',
           '2024-01-11T00:00:00.000000000', '2024-01-12T00:00:00.000000000',
           '2024-01-13T00:00:00.000000000', '2024-01-14T00:00:00.000000000',
           '2024-01-15T00:00:00.000000000', '2024-01-16T00:00:00.000000000',
           '2024-01-17T00:00:00.000000000', '2024-01-18T00:00:00.000000000',
           '2024-01-19T00:00:00.000000000', '2024-01-20T00:00:00.000000000',
           '2024-01-21T00:00:00.000000000', '2024-01-22T00:00:00.000000000',
           '2024-01-23T00:00:00.000000000', '2024-01-24T00:00:00.000000000',
           '2024-01-25T00:00:00.000000000', '2024-01-26T00:00:00.000000000',
           '2024-01-27T00:00:00.000000000', '2024-01-28T00:00:00.000000000',
           '2024-01-29T00:00:00.000000000', '2024-01-30T00:00:00.000000000'],
          dtype='datetime64[ns]')

  • y
    (y)
    float64
    40.5 40.46 40.42 ... 39.58 39.54

    standard_name :

    latitude

    units :

    degrees_north

    array([40.5     , 40.458333, 40.416667, 40.375   , 40.333333, 40.291667,
           40.25    , 40.208333, 40.166667, 40.125   , 40.083333, 40.041667,
           40.      , 39.958333, 39.916667, 39.875   , 39.833333, 39.791667,
           39.75    , 39.708333, 39.666667, 39.625   , 39.583333, 39.541667])

  • x
    (x)
    float64
    -105.7 -105.7 ... -104.8 -104.8

    standard_name :

    longitude

    units :

    degrees_east

    array([-105.75    , -105.708333, -105.666667, -105.625   , -105.583333,
           -105.541667, -105.5     , -105.458333, -105.416667, -105.375   ,
           -105.333333, -105.291667, -105.25    , -105.208333, -105.166667,
           -105.125   , -105.083333, -105.041667, -105.      , -104.958333,
           -104.916667, -104.875   , -104.833333, -104.791667])

• Attributes: (5)

  long_name :

  daily maximum air temperature

  standard_name :

  air_temperature

  units :

  degC

  source_variable :

  PRISM tmax

  is_synthetic :

  0

Pipe · Describe the method without triggering computation¶

In [3]:

from cubedynamics import pipe, verbs as v

lazy_variability = (
    pipe(cube)
    | v.anomaly(dim="time")
    | v.variance(dim="time")
).unwrap()

assert hasattr(lazy_variability.data, "chunks")
lazy_variability

from cubedynamics import pipe, verbs as v lazy_variability = ( pipe(cube) | v.anomaly(dim="time") | v.variance(dim="time") ).unwrap() assert hasattr(lazy_variability.data, "chunks") lazy_variability

Out[3]:

<xarray.DataArray 'tmax' (time: 1, y: 24, x: 24)> Size: 2kB
dask.array<getitem, shape=(1, 24, 24), dtype=float32, chunksize=(1, 12, 12), chunktype=numpy.ndarray>
Coordinates:
  * y        (y) float64 192B 40.5 40.46 40.42 40.37 ... 39.67 39.62 39.58 39.54
  * x        (x) float64 192B -105.7 -105.7 -105.7 ... -104.9 -104.8 -104.8
Dimensions without coordinates: time
Attributes: (12/13)
    long_name:            daily maximum air temperature
    standard_name:        air_temperature
    units:                degC^2
    source_variable:      PRISM tmax
    is_synthetic:         0
    transform_operation:  anomaly
    ...                   ...
    semantic_name:        variance of anomaly of tmax
    semantic_kind:        summary
    semantic_category:    summary
    summary_operation:    variance
    summary_dimensions:   time
    semantic_units:       degC^2

xarray.DataArray

'tmax'

• time: 1
• y: 24
• x: 24

• dask.array<chunksize=(1, 12, 12), meta=np.ndarray>

  Array Chunk Bytes 2.25 kiB 576 B Shape (1, 24, 24) (1, 12, 12) Dask graph 4 chunks in 10 graph layers Data type float32 numpy.ndarray

• Coordinates: (2)
  • y
    (y)
    float64
    40.5 40.46 40.42 ... 39.58 39.54

    standard_name :

    latitude

    units :

    degrees_north

    array([40.5     , 40.458333, 40.416667, 40.375   , 40.333333, 40.291667,
           40.25    , 40.208333, 40.166667, 40.125   , 40.083333, 40.041667,
           40.      , 39.958333, 39.916667, 39.875   , 39.833333, 39.791667,
           39.75    , 39.708333, 39.666667, 39.625   , 39.583333, 39.541667])

  • x
    (x)
    float64
    -105.7 -105.7 ... -104.8 -104.8

    standard_name :

    longitude

    units :

    degrees_east

    array([-105.75    , -105.708333, -105.666667, -105.625   , -105.583333,
           -105.541667, -105.5     , -105.458333, -105.416667, -105.375   ,
           -105.333333, -105.291667, -105.25    , -105.208333, -105.166667,
           -105.125   , -105.083333, -105.041667, -105.      , -104.958333,
           -104.916667, -104.875   , -104.833333, -104.791667])

• Attributes: (13)

  long_name :

  daily maximum air temperature

  standard_name :

  air_temperature

  units :

  degC^2

  source_variable :

  PRISM tmax

  is_synthetic :

  0

  transform_operation :

  anomaly

  analysis :

  reduction_summary

  semantic_name :

  variance of anomaly of tmax

  semantic_kind :

  summary

  semantic_category :

  summary

  summary_operation :

  variance

  summary_dimensions :

  time

  semantic_units :

  degC^2

Figure · Compute only the final spatial answer¶

In [4]:

import matplotlib.pyplot as plt

variability = lazy_variability.compute()
assert not hasattr(variability.data, "chunks")

fig, ax = plt.subplots(figsize=(6.5, 4.5), constrained_layout=True)
variability.plot(ax=ax, cmap="viridis", cbar_kwargs={"label": "Anomaly variance (°C²)"})
ax.set_title("January maximum-temperature variability")
plt.show()

import matplotlib.pyplot as plt variability = lazy_variability.compute() assert not hasattr(variability.data, "chunks") fig, ax = plt.subplots(figsize=(6.5, 4.5), constrained_layout=True) variability.plot(ax=ax, cmap="viridis", cbar_kwargs={"label": "Anomaly variance (°C²)"}) ax.set_title("January maximum-temperature variability") plt.show()

What the figure tells us¶

The final map identifies locations with more variable departures while the intermediate anomaly cube remained lazy.

Try the next variation¶

Change chunk sizes and confirm that final values remain identical.

Data used¶

Field	Frozen analysis input
Provider	PRISM Group, Oregon State University
Product	AN91d daily 4 km time series
Dates	2024-01-01 to 2024-01-30
Fixture	tests/fixtures/real_data/prism_boulder_january_2024.nc
Provenance record	tests/fixtures/real_data/prism_boulder_january_2024.provenance.json

The PRISM source reference describes current catalog support; the fixture record above identifies the observations used here. Data validation documents checksums and acceptance checks. The analytical baseline and thresholds belong to this story, not the provider.

Reproduce¶

Clone the repository, then run these commands from its root:

python -m pip install -e ".[vignettes]"
python scripts/run_vignettes.py docs/vignettes/lazy_composition.ipynb

No network is needed after installation. Open the downloaded notebook in Jupyter and run all cells to see the same figures. The website executes these cells during its strict build. Environment setup and the vignette contract explain the workflow. The first code cell prints cd.version_info() so a rendered result can be tied to a package path and, for development checkouts, a Git commit.

See also¶

temperature · precipitation · anomaly · variance

Learn the grammar · All vignettes