Getting started

Requirements

COSIPY is compatible with Python 3.9-3.12 on Linux and MacOS. If you think your specific Python version or operating system causes an issue with COSIPY, please create a topic in the forum. The model is tested and developed on:

Python 3.9-3.12 on Ubuntu 22.04 / Debian 12

Anaconda distribution on macOS

Anaconda 3 64-bit (Python 3.9) on CentOS Linux 7.4

High-Performance Cluster Erlangen-Nuremberg University

Warning

COSIPY 2.0 is not backwards-compatible with older versions of COSIPY. Please read the instructions on upgrading from an older version, and using the new configuration system.

Installation

For Python 3.11 and above, the recommended environment manangers are conda/mamba.

Pre-requisites

Install GDAL:

sudo apt-get install gdal-bin libgdal-dev
pip install --upgrade gdal==`gdal-config --version` pybind11  # with pip
conda install gdal  # with conda/mamba

If you are installing dependencies with conda/mamba, use -c conda-forge if it does not already have the highest channel priority.

When you are installing from source, the provided makefile will install the gdal package automatically.

Note

If you are installing with pip and Python 3.11+, you will need to compile and install richdem. This is not necessary when using conda/mamba.

The icc_rt package may provide a performance boost on some systems:

pip install icc-rt             # with pip
conda install icc_rt -c numba  # with conda/mamba

Installation with pip

Installing COSIPY as a package allows it to run from any directory:

pip install cosipymodel
conda install richdem  # if you are using conda/mamba and python 3.11+
cosipy-setup  # generate sample configuration files
cosipy-help   # view help

This is the recommended installation method if you do not plan to modify the source code.

Installation from Source

Activate your preferred python environment, then install dependencies:

git clone https://github.com/cryotools/cosipy.git
cd cosipy

make install-conda-envs                      # install using conda/mamba
conda install --file conda_requirements.txt  # install with conda

pip install -r requirements.txt              # install default environment
pip install -r dev_requirements.txt          # install dev environment

python3 COSIPY.py -h
make commands      # if you prefer less typing
make setup-cosipy  # generate configuration files

Installation as an Editable

Installing COSIPY as an editable allows it to run from any directory.

The provided makefile can simplify your workflow. View all possible commands using make help.

git clone https://github.com/cryotools/cosipy.git
cd cosipy

make install            # with conda/mamba
make install-pip        # with pip

cosipy-setup            # generate sample configuration files
cosipy-help             # view help

That’s it! Other installation options with pip:

pip install -e .        # identical to make install-pip
make install-pip-tests  # install with test dependencies using pip
make install-pip-dev    # install with development dependencies using pip

cosipy-setup            # generate sample configuration files
cosipy-help             # view help

Upgrading from an Older Version of COSIPY

COSIPY 2.0 is not backwards-compatible with previous versions of COSIPY. If you have written your own modules that import from constants.py, config.py, or use Slurm, these will break.

Navigate to COSIPY’s root directory and convert your existing configuration files:

pip install toml
git fetch --all
git checkout master -- convert_config.py
python convert_config.py  # convert .toml files

This works on any branch regardless of local changes. Alternatively you can copy and run convert_config.py into any older COSIPY source tree. This will preserve your configuration for config.py, constants.py, aws2cosipyConfig.py and wrf2cosipyConfig.py.

Warning

Parameters for create_static must still be added manually to the generated utilities_config.toml. Custom configuration variables that do not appear in the main branch must also be added manually.

Checkout a new branch with a clean version of COSIPY and merge your modifications.

git checkout master
git pull
git checkout -b <new-branch-name>
git merge --no-ff <old-branch-name>  # Good luck!

You can also merge the new version of COSIPY into an existing branch, but this creates even more merge conflicts. Most conflicts involve importing configuration parameters or constants. In most cases you simply need to prepend Config. or Constants. to a variable. Please read the documentation for the new configuration system.

After updating to the latest version of COSIPY, run python COSIPY.py --help to see how to specify paths to configuration files. COSIPY will default to ./config.toml, ./constants.toml, ./slurm_config.toml, ./utilities_config.toml in the current working directory. You no longer need to hardcode different simulation parameters into a single file.

Tutorial

For this tutorial, download or copy the sample data folder and place it in your COSIPY working directory. If you have installed COSIPY as a package, use the entry point setup-cosipy to generate the sample configuration files. Otherwise, run make setup-cosipy.

Pre-Processing

COSIPY requires a file with the corresponding meteorological and static input data. Various tools are available to create the file from simple text or geotiff files.

Create the static file

In the first step, topographic parameters are derived from a Digital Terrain Model (DEM) and written to a netCDF file. A shape file is also required to delimit the glaciated areas. The DEM and the shapefile should be in lat/lon WGS84 (EPSG:4326) projection.

Note

The DEM can be reprojected to EPSG:4326 using gdal:

gdalwarp -t_srs EPSG:4326 dgm_hintereisferner.tif dgm_hintereisferner-lat_lon.tif

COSIPY comes with the script create_static_file.py located in the utilities folder. This script runs some gdal routines in the command line. At the moment this is only compatible with UNIX and MacOS. The script creates some intermediate netCDF files (dem.nc, aspect.nc, mask.nc and slope.nc) that are automatically deleted after the static file is created.

Open utilities_config.toml. Under create_static.paths, check the paths point to the DEM n30_e090_3arc_v2.tif (SRTM) and the shapefile Zhadang_RGI6.shp provided in the ./data/static/ folder.

The static file is created using either:

make create-static  # from source
python -m cosipy.utilities.createStatic.create_static_file  # from source
cosipy-create-static  # from entry point

The command creates a new file Zhadang_static.nc in the ./data/static/ folder.

Create the COSIPY input file

Creating the input file requires the static information file from the previous section. To convert the data from an automatic weather station (AWS) use the conversion script aws2cosipy.py, located in the folder ./utilities/aws2cosipy/. A sample configuration is available in utilities_config.toml which defines the structure of the AWS file and other user-defined parameters. Since the input file provides point information, the data is interpolated via lapse rates for two-dimensional runs. The solar radiation fields are based on a model by Wohlfahrt et al. (2016). Other variables like wind velocity and cloud cover fraction are assumed constant over the domain.

Note

The script aws2cosipy.py is only an illustration of how data can be prepared for COSIPY. For most applications it is recommended to develop your own data interpolation routines.

The script is executed with:

# from source
python -m cosipy.utilities.aws2cosipy.aws2cosipy \
    -i ./data/input/Zhadang/Zhadang_ERA5_2009_2018.csv \
    -o ./data/input/Zhadang/Zhadang_ERA5_2009.nc \
    -s ./data/static/Zhadang_static.nc \
    -b 20090101 -e 20091231

# from entry point
cosipy-aws2cosipy \
    -i ./data/input/Zhadang/Zhadang_ERA5_2009_2018.csv \
    -o ./data/input/Zhadang/Zhadang_ERA5_2009.nc \
    -s ./data/static/Zhadang_static.nc \
    -b 20090101 -e 20091231

If the script executes successfully it will create the file ./data/input/Zhadang/Zhadang_ERA5_2009.nc.

Usage:

cosipy.utilities.aws2cosipy [-h] [-u <path>] -c <path> -o <path> -s <path> [-b <str>] [-e <str>] [-xl <float>] [-xr <float>] [-yl <float>] [-yu <float>]

Required arguments:

-i, --csv_file <path>: Path to .csv file with meteorological data.
-o, --cosipy_file <path>: Path to the resulting COSIPY netCDF file.
-s, --static_file <path>: Path to static file with DEM, slope etc.

Optional arguments:

-u, --utilities <path>: Relative path to utilities’ configuration file.
-b, --start_date <str>: Start date.
-e, --end_date <str>: End date.
--xl <float>: Left longitude value of the subset.
--xr <float>: Right longitude value of the subset.
--yl <float>: Lower latitude value of the subset.
--yu <float>: Upper latitude value of the subset.

Run the COSIPY model

To run COSIPY, run the following command in the root directory:

python COSIPY.py  # from source
run-cosipy        # from package

The example should take about a minute on a workstation with 4 cores.

Running COSIPY

COSIPY accepts arguments specifying paths to configuration files.

Usage:

python COSIPY.py [-h] [-c <path>] [-x <path>] [-s <path>]  # from source
run-cosipy [-h] [-c <path>] [-x <path>] [-s <path>]  # from package

Optional arguments:

-c <path>, --config <path>: Relative path to configuration file.
-x <path>, --constants <path>: Relative path to constants file.
-s <path>, --slurm <path>: Relative path to Slurm configuration file.

Entry Points

If installed as an editable or package, COSIPY provides several entry points to speed up common operations. These entry points accept python arguments (such as --help).

Available shortcuts:

cosipy-help:: Display help for running COSIPY.
cosipy-shortcuts:: Display available entry points.
cosipy-setup:: Setup missing configuration files.
cosipy-run:: Run COSIPY. Accepts python arguments.
cosipy-aws2cosipy:: Convert AWS data to netCDF.
cosipy-create-static:: Create static file.
cosipy-wrf2cosipy:: Convert WRF data to netCDF.
cosipy-plot-field:: Generate field plots.
cosipy-plot-profile:: Generate profile plots.
cosipy-plot-vtk:: Generate 3D plots.
help-cosipy:: Alias for cosipy-help.
run-cosipy:: Alias for cosipy-run.
setup-cosipy:: Alias for cosipy-setup.

Makefile

The provided makefile can simplify your workflow. View all possible commands using make help.

Available shortcuts:

black:: Format all python files with black.
build:: Build COSIPY package.
commands:: Display help for COSIPY.
commit:: Test, then commit.
coverage:: Run pytest with coverage.
docs:: Build documentation.
flake8:: Lint with flake8.
format:: Format all python files.
help:: Display this help screen.
install-conda-env:: Install conda/mamba dependencies.
install:: Install editable package using conda/mamba.
install-pip-all:: Install editable package with tests & documentation using pip.
install-pip-dev:: Install editable package in development mode using pip.
install-pip-docs:: Install editable package with local documentation using pip.
install-pip:: Install editable package using pip.
install-pip-tests:: Install editable package with tests using pip.
isort:: Optimise python imports.
pkg:: Run tests, build documentation, build package.
pylint:: Lint with Pylint.
run:: Alias for make commands.
setup-cosipy:: Generate COSIPY configuration files.
tests:: Run tests.

Configuration

Note

Configure parameters/constants in config.toml, constants.toml, slurm_config.toml and utilities_config.toml.

All user configuration is done with .toml files. If COSIPY is installed as a package, generate sample configuration files using setup-cosipy. Configuration is split into four parts: model configuration, constants, utilities, and Slurm configuration. You can keep multiple configuration files for different simulations in the same (or indeed any working directory).

If you are using a cluster, you can also chain multiple simulations with a single batch script:

python $WORK/COSIPY.py -c config_01.toml -x constants_01.toml -s slurm_01.toml
python $WORK/COSIPY.py -c config_02.toml -x constants_02.toml -s slurm_02.toml

Select which output variables are saved to disk under [OUTPUT_VARIABLES] in config.toml. This can prevent out-of-memory errors when working with very large datasets. This replaces the cosipy/output and cosipy/output.full files.

Import configuration parameters or constants into a module. These are read-only to avoid namespace collisions.

from cosipy.config import Config
from cosipy.constants import Constants

foo = Config.foo  # declare at module level if used in an njitted function

@njit
def get_foo_njit(...):
    """Njitted functions cannot reference the imported parameters directly."""
    return foo

def get_foo_nopython(...):
    """Non-compiled functions can reference the parameters directly."""
    return Config.foo