Skip to content

Reproduce the Data Lake

The Fire VASE data lake is the manuscript-scale handoff package. It contains the source fire and climate caches, Parquet lakehouse tables, developmental morphology outputs, schemas, figures, manuscripts, and release manifests needed to reproduce the analysis. Its scientific role is to preserve the chain from ordered FIRED observations to a comparable developmental response before weather or other explanatory layers are attached.

Get the Shared Data Lake

The current shared folder is:

CyVerse Fire_Vase

Expected local package path:

data_lake/fire-vase-data-lake-v0.1

After downloading or syncing the package, verify every file against the release checksums:

uv run python scripts/check_reproducibility.py

This reads the full package, so it can take several minutes.

Inspect Or Package A Local Lake

The release definition is config/data_release.yml.

Create or refresh the manifest and checksums:

uv run python scripts/prepare_data_lake.py --mode manifest --checksum

Create a materialized handoff directory with hardlinks:

uv run python scripts/prepare_data_lake.py --mode hardlink --checksum

Create an independent copy for upload or external transfer:

uv run python scripts/prepare_data_lake.py --mode copy --checksum

The packager writes:

  • manifest.json
  • file_manifest.csv
  • checksums.sha256
  • README.md
  • files/, for hardlink and copy modes

Rebuild From Source

Use this path when you want to recreate the lakehouse rather than start from the shared data lake.

  1. Install the repository.
uv sync
  1. Cache the gridMET variables used by the manuscript.
uv run python scripts/cache_gridmet_years.py --preset comprehensive --keep-going
  1. Build the full Fire VASE lakehouse tables.
uv run python scripts/fire_vase_lakehouse_pilot.py \
  --config config/fire_vase_pipeline.yml \
  --output-root scratch/fire_vase_run_full \
  --full-population
  1. Build centroid and perimeter climate attribution tables.
uv run python scripts/fire_vase_build_climate_tables.py \
  --include-optional-variables \
  --table-root scratch/fire_vase_run_full/tables

uv run python scripts/fire_vase_build_perimeter_climate_tables.py \
  --include-optional-variables
  1. Build developmental morphology and climate-coupling products.
uv run python scripts/fire_vase_developmental_morphology_analysis.py \
  --table-root scratch/fire_vase_run_full/tables \
  --data-output-dir scratch/fire_vase_developmental_morphology
  1. Regenerate the climate-revision analysis products and manuscript figures.
uv run python scripts/fire_vase_climate_revision.py
  1. Refresh the data-lake package metadata.
uv run python scripts/prepare_data_lake.py --mode manifest --checksum