Documents · API reference

Data and source configuration

Noun definitions and source facts live in Library. This page documents discovery, loading, provenance, schema and QA functions. Signatures and descriptions below come from the runtime docstrings.

Discovery

Source discovery metadata for CubeDynamics scientific nouns.

The catalog is intentionally small and declarative. It describes source flavors that are actually wired into the package; planned sources do not appear here until their loaders, tests, QA, and documentation exist.

list_sources()

Return implemented source flavors grouped by scientific noun.

sources(noun)

Return implemented source flavors for noun.

Planned integrations are deliberately omitted. An unknown noun raises a message that lists the vocabulary currently available.

describe(noun, source=None)

Return human-readable metadata for an implemented noun/source flavor.

Private and local storage

Read project-owned cubes, geometry layers and batched tables through the filesystem without public catalog registration. See Private data on HPC scratch.

Private cubes, features and batched tables on local or mounted storage.

open_local_cube(paths, *, dataset_id, revision=None, variables=None, chunks=None, engine=None)

Open private NetCDF files or one Zarr store with deferred data reads.

Parameters:

Name	Type	Description	Default
paths	path or sequence of paths	A NetCDF file, explicit file list, local glob, or a single directory ending in .zarr. Paths must be visible to the Python process, including mounted HPC scratch. No SSH connection or transfer is made.	required
dataset_id	str	Project-owned dataset identity, for example lab/soil-moisture.	required
revision	str	User-supplied version or acquisition identity. No version is inferred from a scratch filename; an omitted revision remains unspecified.	None
variables	sequence of str	Data variables to retain. Coordinate metadata is preserved.	None
chunks	mapping	Native dimension names to Dask chunk sizes. Default uses backend chunks; unchunked NetCDF may need explicit sizes to bound task memory.	None
engine	str	xarray NetCDF engine. Zarr uses its own backend; install cubedynamics[local] for Zarr support.	None

Returns:

Type	Description
Dataset	Dask-backed data with native dimensions, units, CRS and attributes. Multiple NetCDF files combine by coordinates with exact non-concat alignment. No regridding, coordinate renaming, unit conversion, publication, or scientific certification is performed.

Notes

Opening reads metadata and coordinate indexes. Computation reads the source chunks directly; it does not copy the dataset into a package cache. Keep files unchanged and accessible until computation finishes, then call close() (or use the Dataset as a context manager). Distributed workers need the same paths and permissions. Scratch storage may be purged.

local_source_provenance is JSON containing resolved paths, dataset ID, declared revision and file size/mtime (directory mtime for Zarr). These are discovery records, not content checksums or an immutable snapshot. Existing scientific provenance and synthetic flags are retained without inventing an observational or publication status. Paths may be private; review metadata before sharing results.

open_local_features(path, *, dataset_id, revision=None, layer=None, crs=None, max_features=100000)

Read a private geometry layer as an eager GeoPandas GeoDataFrame.

Parameters:

Name	Type	Description	Default
path	path	One local GeoJSON (.geojson/.json), GeoPackage (.gpkg), or GeoParquet (.parquet/.geoparquet) file, visible to this Python process.	required
dataset_id	str	Project-owned dataset identity.	required
revision	str	Declared acquisition/version identity, not inferred from the filename.	None
layer	str	Required named layer for GeoPackage; unsupported for other formats.	None
crs	str	Explicit CRS when the file lacks one. If already declared, it must agree. This never overrides a different CRS or reprojects coordinates.	None
max_features	int	Positive row limit (default 100000). GeoJSON/GeoPackage reads request at most limit + 1 rows to detect overflow; GeoParquet counts come from metadata before loading. This is a feature-count, not a byte budget.	100000

Returns:

Type	Description
GeoDataFrame	Native point, polygon, multipart or other geometry and attribute columns, with a known CRS. Empty layers, missing/empty/invalid geometry, unknown/conflicting CRS and excess rows raise errors; no repair or rasterization is implicit. This read materializes the selected layer. Prefer spatially partitioned layers when a whole layer is too large.

Notes

Uses the same filesystem visibility and JSON provenance policy as open_local_cube. Units and observation dates remain source columns; project-specific validation and explicit alignment/joins are still needed. No public registration, transfer or certification is performed.

open_local_table(path, *, dataset_id, revision=None, columns=None, batch_size=100000, dtype=None)

Iterate through a private CSV, TSV or Parquet file in row batches.

Parameters:

Name	Type	Description	Default
path	path	One local .csv, .tsv or .parquet file. No implicit remote access.	required
dataset_id	str	Project-owned dataset identity.	required
revision	str	Declared acquisition/version identity.	None
columns	sequence of str	Columns to read, in the requested order; missing names raise errors.	None
batch_size	int	Positive maximum rows per returned DataFrame (default 100000). CSV/TSV uses pandas chunks; Parquet uses Arrow record batches with column projection. Variable-length values can still be large; this is not a byte budget. Empty files with a header yield no data batches.	100000
dtype	mapping	Explicit pandas CSV/TSV dtypes, e.g. {"site_id": "string"} to preserve leading zeros. Parquet uses Arrow-backed pandas dtypes to retain nullable stored types and rejects this option. Dates, units, keys and coordinates are never inferred scientifically.	None

Returns:

Type	Description
iterator of pandas.DataFrame	Measurement reads occur during iteration. Each batch has the same local_source_provenance JSON attribute as the cube/feature loaders. Use contextlib.closing if iteration may stop early, or explicitly close the iterator. Exhaustion closes the underlying reader. Keep source files fixed until iteration finishes. GeoParquet must use open_local_features so geometry/CRS metadata is not silently lost.

Notes

Pandas performs ordinary CSV type inference unless dtype is supplied. No join, deduplication, geometry construction or xarray conversion occurs. Parquet requires the local extra. The iterator is local and synchronous; it is not a distributed Dask DataFrame or a public-catalog registration.

Temporal support

Time labels and physical observation intervals are separate metadata. See the Temporal alignment guide.

Temporal-support metadata and diagnostics for scientific cubes.

Time coordinates are labels. The observations attached to those labels may represent instants or intervals, and two equal labels need not describe the same interval in the physical world. This module keeps that distinction metadata-only: it never shifts, resamples, aggregates, or computes cube data.

TemporalSupport(resolution=None, support_type='unknown', label_convention=None, reference_timezone=None, start_offset=None, end_offset=None, known=False, evidence=None) dataclass

Source-qualified meaning of one object's temporal observations.

instant property

Whether the support is a known point observation.

interval property

Whether the support is a known observation interval.

as_dict()

Return a serializable representation using public metadata names.

scientifically_equal(other)

Return exact support equality, or None when either is unknown.

TemporalAlignmentReport(coordinates, temporal_support, left, right, left_source=None, right_source=None) dataclass

Metadata-only comparison of coordinate labels and temporal support.

exact property

Whether both coordinate labels and known supports are exact.

as_dict()

Return a stable, serializable diagnostic record.

temporal_support(obj)

Inspect temporal-support metadata without touching array values.

Missing metadata is represented as an unknown support, never inferred from a daily-looking coordinate alone.

observation_intervals(obj, *, time_dim='time')

Derive one-dimensional observation starts and ends from support rules.

Only the time coordinate is materialized. Cube values are neither copied nor computed, so a Dask-backed input remains lazy and unchanged.

compare_temporal_support(left, right, *, time_dim='time')

Compare time-coordinate labels separately from observation support.

Source-specific loaders

Provider-specific loaders remain supported alongside scientific nouns. Compare implemented source flavors before choosing.

Load a GRIDMET-like climate cube.

Parameters:

Name	Type	Description	Default
lat	float	Latitude/longitude of a point of interest. When provided, a small bounding box is generated around the point so that GRIDMET pixels are fetched for the surrounding area.	None
lon	float	Latitude/longitude of a point of interest. When provided, a small bounding box is generated around the point so that GRIDMET pixels are fetched for the surrounding area.	None
bbox	sequence of float	Bounding box defined as [min_lon, min_lat, max_lon, max_lat].	None
aoi_geojson	mapping	GeoJSON Feature/FeatureCollection describing the area of interest. A bounding box is derived from the geometry.	None
start	datetime - like	Temporal extent for the request.	None
end	datetime - like	Temporal extent for the request.	None
variable	str	GRIDMET variable to request. variables may be used to explicitly pass a list.	None
freq	str	Temporal frequency code. freq overrides time_res when set and defaults to monthly ("MS") to mirror the documentation.	None
time_res	str	Temporal frequency code. freq overrides time_res when set and defaults to monthly ("MS") to mirror the documentation.	None
chunks	mapping	Custom Dask chunk mapping.	None
prefer_streaming	bool	Retained for compatibility. Both real paths read the authoritative annual gridMET NetCDF assets; the preferred path preserves Dask chunks.	True
show_progress	bool	Display retrieval progress when available. Set to False to disable progress reporting.	True

Notes

The modern API requires keyword arguments and exactly one AOI specification (lat/lon, bbox or aoi_geojson). Legacy positional usage of the form load_gridmet_cube(variable, start, end, aoi, ...) is still supported but deprecated.

Load a PRISM climate cube.

Parameters:

Name	Type	Description	Default
lat	float	Latitude/longitude of a point of interest. When provided, a small bounding box is generated around the point so that PRISM pixels are fetched for the surrounding area.	None
lon	float	Latitude/longitude of a point of interest. When provided, a small bounding box is generated around the point so that PRISM pixels are fetched for the surrounding area.	None
bbox	sequence of float	Bounding box defined as [min_lon, min_lat, max_lon, max_lat].	None
aoi	mapping	Backward-compatible alias for bbox as a mapping containing min_lon, min_lat, max_lon, and max_lat.	None
aoi_geojson	mapping	GeoJSON Feature/FeatureCollection describing the area of interest. A bounding box is derived from the geometry.	None
start	datetime - like	Temporal extent for the request.	None
end	datetime - like	Temporal extent for the request.	None
variable	str or sequence of str	PRISM variable(s) to request. variables may also be used for clarity when passing multiple entries. When a single variable is requested through variable, the loader returns an xarray.DataArray.	"ppt"
time_res	str	Temporal frequency code. freq overrides the compatibility name time_res. Keyword-style real requests default to daily "D"; current real streaming supports daily data only and does not silently aggregate daily observations to monthly values.	None
freq	str	Temporal frequency code. freq overrides the compatibility name time_res. Keyword-style real requests default to daily "D"; current real streaming supports daily data only and does not silently aggregate daily observations to monthly values.	None
chunks	mapping	Custom Dask chunk mapping.	None
prefer_streaming	bool	Whether to use lazy, server-side AOI subsets from the PRISM THREDDS mirror. Daily slices are fetched only when Dask computes them.	True
show_progress	bool	Display a progress bar while synthetic demo data are generated. Real streaming remains lazy and reports progress through the Dask scheduler.	True

Notes

The modern API requires keyword arguments and exactly one AOI specification (lat/lon, bbox or aoi_geojson). Legacy positional usage of the form load_prism_cube(variables, start, end, aoi, ...) is still supported but deprecated.

Real streaming currently supports daily ppt, tmean, tmin, and tmax. Synthetic fallback is disabled unless allow_synthetic=True.

Sentinel-2 data access helpers.

load_s2_cube(lat, lon, start, end, edge_size=1028, resolution=10, cloud_lt=40, bands=None, chunks=None)

Stream Sentinel-2 L2A data via cubo and return a dask-backed xarray object.

load_s2_ndvi_cube(lat, lon, start, end, edge_size=1028, resolution=10, cloud_lt=40, bands=None, chunks=None)

Stream Sentinel-2 and return an NDVI cube ready for downstream ops.

Serving revisions

Source-controlled serving history and safe promotion/rollback queries.

serving_history(noun, source_flavor)

Return immutable history entries for one noun/source pair.

current_revision_record(noun, source_flavor)

Return the single current revision, rejecting ambiguous history.

validate_promotion(candidate, *, certification_outcome)

Validate a proposed promotion without rewriting source-controlled history.

rollback_target(noun, source_flavor)

Return the newest validated retired revision eligible for rollback.

Lifecycle decisions and records

Small source-lifecycle value objects used by the existing data catalog.

These types describe source maintenance and certification without changing the public noun-and-pipe programming model. The authoritative noun/source records remain in :mod:cubedynamics.data.catalog.

ServingRevision(noun, source_flavor, created, sequence) dataclass

Immutable identifier for one CubeDynamics interpretation of a source.

parse(value) classmethod

Parse noun.source@YYYY-MM-DD.N and reject ambiguous identifiers.

ServingRevisionRecord(revision_id, stage, status, created_at, promoted_at=None, adapter_version=None, schema_fingerprint=None, qa_evidence=None, normalization_contract=None, caveats=()) dataclass

Durable metadata that links an immutable revision to its evidence.

UpstreamIdentity(provider, product, endpoint, strategy, observed=dict(), retrieved_at=None) dataclass

Provider-native identity observed for one bounded retrieval.

CertificationRecord(mode, outcome, gates, serving_revision, last_validated, evidence=dict(), caveats=()) dataclass

Machine-readable evidence summary for offline or live source QA.

decide_source_change(change, *, source_mode)

Translate a classified provider change into a deterministic response.

Schema normalization

Deterministic scientific schema descriptions for xarray source results.

normalize_xarray_schema(value)

Return scientifically relevant structural metadata in canonical order.

Array values, chunk shapes, retrieval timestamps, and dimension sizes are deliberately excluded so equivalent bounded requests share a fingerprint. Variable dimension order is retained because it can affect interpretation.

normalize_vector_schema(*, fields, geometry_type, crs, layer_id=None, coded_domains=None)

Normalize a vector schema without requiring a geospatial dependency.

normalize_api_schema(*, fields, units=None, parameter_ids=None, geography_ids=(), datetime_representation)

Normalize an API/tabular response contract for later source families.

schema_fingerprint(value)

Hash the canonical scientific schema without reading array values.

fingerprint_normalized_schema(normalized)

Hash any normalized schema representation deterministically.

compare_normalized_schemas(expected, observed)

Return deterministic, path-level schema drift evidence.

QA and certification

Reusable structural QA profiles for CubeDynamics source integrations.

Profiles evaluate scientific structure and metadata. Source-specific QA may add range, physics, checksum, or visual checks, but it should reuse one of these profiles instead of inventing a second source-registration system.

list_qa_profiles()

Return the stable names of the reusable QA profiles.

get_qa_profile(name)

Return one profile, naming available choices on error.

evaluate_qa_profile(name, value, *, context=None, caveats=())

Evaluate a reusable profile and return an explicit certification outcome.

Shared live-source certification built on the reusable QA profiles.

certify_live_sample(sample, *, qa_profile, serving_revision, endpoint_verified, bounded_access_verified, upstream_identity_verified, expected_schema=None, caveats=())

Certify one tiny remote sample with the same profile used offline.

blocked_live_certification(*, serving_revision, reason)

Represent unavailable credentials/services without inventing a pass.

write_live_certification(result, output)

Persist health/certification evidence as an uploadable JSON artifact.

See also

Source lifecycle contract · Source QA evidence · Legacy semantic helpers · Provenance lesson