Add 0.25 degree model configurations

[aidanheerdegen]

Overview

The full ACCESS-OM2 release requires the 0.25 degree configurations, IAF and RYF atmospheric forcing, with physics only and bgc.

The goal is to have all four COSIMA 0.25 configurations as specially named branches in this repository, versioned (2.0) and tagged:

• release-025deg_jra55_ryf (repo)
• release-025deg_jra55_iaf (repo)
• release-025deg_jra55_ryf_bgc (repo)
• release-025deg_jra55_iaf_bgc (repo)

For each configuration this requires the following steps:

1. Add configuration to this repository in a dev-* branch
2. git checkout an update branch from this dev-* branch, modify it to use input paths from vk83, ACCESS-NRI deployed model versions and update some metadata and configuration options
3. Make a PR to update the dev-* branch created above. Ensure QA checks pass, then merge
4. Create a release-* branch if one doesn't already exist from the dev-* branch and run a special workflow to add reproducibility checksums, version and tag.

The release-1deg_jra55_ryf branch is the template to follow for RYF configurations. Similarly the release-1deg_jra55_iaf branch is the template to follow for IAF configurations.

There is a Contributing Guide which covers in more detail how to add and change configurations, but below are specific instructions for this specific task of porting COSIMA configurations to this repo.

The ACCESS-OM2 spack.location file gives the paths to the installed components:

cice5@git.2023.10.19=2023.10.19         /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo
libaccessom2@git.2023.10.26=2023.10.26  /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo
mom5@git.2023.11.09=2023.11.09          /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz

The ACCESS-OM2-BGC spack.location file gives the paths to the BGC installed components (note only the MOM5 build differs from the physics only ACCESS-OM2):

cice5@git.2023.10.19=2023.10.19         /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo
libaccessom2@git.2023.10.26=2023.10.26  /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo
mom5@git.2023.11.09=2023.11.09          /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-64l5azdtcoxhrgb5ynn2vued5lmjvn33

Steps to follow to add a COSIMA configuration to this repo:

1. Clone this repo, e.g. gh repo clone ACCESS-NRI/access-om2-configs
2. Clone configuration repo, e.g. gh repo clone COSIMA/025deg_jra55_ryf
3. cd access-om2-configs
4. Add the COSIMA repo as a remote, e.g. remote add 025deg_jra55_ryf ../025deg_jra55_ryf
5. Create branch from config repo, e.g. git checkout -b dev-025deg_jra55_ryf 025deg_jra55_ryf/master
6. Add configuration to GitHub repository, e.g. `git push dev-025deg_jra55_ryf

Steps to modify a configuration:

Because this is modifying an existing COSIMA configuration we would like to make sure the changes we're making are not introducing any source of error. There is a slow way (test bit reproducibility) or a fast way (use manifest checksums). Both options are outlined below. Choose which you're comfortable using. If you do choose the slow way, it probably isn't necessary to do it for all configurations, as it is possible to compare the branches and use that as a sanity check, as they should differ in small but deterministic ways.

1. Checkout update branch (don't choose a name that begins with dev- or release-), e.g. git checkout update-1deg_jra55_ryf

---

Either Option 1: make changes and rely on manifests for error checking (fast)

1. Modify config.yaml input paths to point to the new structure in vk83 (see 1 degree config). Note paths to qv56 for IAF configs remains unchanged.
2. Run payu setup to generate new input manifests. Run git diff manifests/input.yaml and confirm that only fullpath and binhash entries modified. All md5 hashes should be unchanged. This confirms the inputs are identical to the COSIMA configuration.
3. Modify model executable paths to those in vk83 (see above for paths, and note there are different CICE executables for the different resolutions)
4. Run payu setup to generate new executable manifests. Run git diff manifests/input.yaml and confirm that fullpath, binhash and md5 entries are all modified. All md5 hashes should be changed.

Or Option 2: make changes and use model runs to check reproducibility (or close to it)

1. Set runlog: false to prevent unnecessary git commits during testing
2. Change the run length to something short for testing purposes (maybe 1 day for 0.25 degree)
3. Create a ground-truth from the COSIMA configuration to compare against. Make sure you have diagnostic output at a high enough frequency that fields are output for the shortened run length. Good fields to choose are prognostic variables like salt, temp and velocities u, v (alternatively can compare the restart files the model outputs).
4. Modify config.yaml input paths to point to the new structure in vk83 (see 1 degree config). Note paths to qv56 for IAF configs remains unchanged.
5. Run short test run and check for reproducibility (see below). This confirms we have the same inputs as the COSIMA configuration.
6. Modify model executable paths to those in vk83 (see above for paths, and note there are different CICE executables for the different resolutions)
7. Do short test run and check differences are small (should just be some emerging noise depending on run length)
8. Change run length back, and turn runlog back on

---

2. Create a tools subdirectory and git mv resub.sh tools/.. Update config.yaml as required to point to new locations.
3. Change collate exe to point to /g/data/vk83/apps/mppnccombine-fast/0.2/bin/mppnccombine-fast
4. Update README (see 1 degree config example)
5. Add summary of testing done to this issue
6. git push update branch to repository and create a pull request to the dev-* branch you created and pushed in first section.
7. Fix all QA tests which fail in the CI testing. See Contributing Guide for an explanation of some the tests and what is typically required for a conforming configuration.
8. Once all QA tests are passing, request a review. When PR is approved merge.
9. Create a release branch and add reproducibility checksums following the instructions. In this case specify the initial version to be 2.0 to indicate that this is no longer bit reproducible with the COSIMA (1.0) versions.

As a guide this GitHub comparison shows the changes required for the 1 degree RYF configuration.

224b125...3f4cd6e

Bit reproducibility comparisons

We no longer have bit reproducibility with historical (COSIMA) versions of ACCESS-OM2 because we have opted to turn on compile options to ensure we have reproducibility across modified restart boundaries.

However the outputs should remain bit reproducible until the point where executables are modified.

Regardless, even when the executables are modified it is a good idea to run the model to check the differences after a short time are what we'd expect: just some noise in the fields.

For comparing model outputs I recommend a few different tools:

• nccmp (/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/nccmp)

nccmp is silent when files are equivalent, otherwise it will echo to STDERR whether metadata or a specific variable differs. By default, comparing stops after the first difference.

• ncdiff (/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/ncdiff)

ncdiff subtracts one netcdf file from another

• ncview (/apps/ncview/2.1.7/bin/ncview)

Simple Xwindows based netCDF file viewer. When subtracting one file from another it is a good idea to make sure the files contain actual data. If they're empty you may get a false negative for bit reproducibility as they will be identical.

6/4: Instructions updated

[aidanheerdegen]

There is a topography update that we may want to wait until it is added.

COSIMA/access-om2#265

And an issue with misaligned ocean/ice grids has recently been spotted

COSIMA/access-om2#280

We can go ahead and make the configurations now and update them when fixes are available, or wait until the issues are ironed out. Any comment @aekiss?

[aekiss]

• The topography has now been updated (IAF and RYF, master and master+ngc) - probably should tag this?
• Grid fix is underway.

[aidanheerdegen]

Awesome. Thanks.

[aidanheerdegen]

The BGC input files will need to be copied over, c.f. 1 degree model

#19 (comment)

[aidanheerdegen]

Input files copied into vk83 directory tree

$ tree /g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.025deg/
/g/data/vk83/experiments/inputs/access-om2/ice/initial_conditions_biogeochemistry/global.025deg/
└── 2022.02.24
    ├── i2o.nc
    ├── o2i.nc
    └── README

1 directory, 3 files

$ tree /g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.025deg/
/g/data/vk83/experiments/inputs/access-om2/ocean/biogeochemistry/global.025deg/
└── 2022.02.24
    ├── bgc_param.nc
    ├── co2_iaf.nc
    ├── co2_ryf.nc
    ├── csiro_bgc.res.nc
    ├── csiro_bgc_sediment.res.nc
    ├── dust.nc
    └── README

1 directory, 7 files

[aidanheerdegen]

Current plan is to defer release of 0.25 degree configurations until updated an updated ice grid is available and a model verification experiment has been completed.

The release-025deg_jra55_ryf branch has been renamed dev-025deg_jra55_ryf to reflect it's status as a development configuration.

[aidanheerdegen]

Current plan is to defer release of 0.25 degree configurations until updated an updated ice grid is available and a model verification experiment has been completed.

Plans have changed. 0.25 degree will be part of this release.

[anton-seaice]

release-025deg_jra55_iaf, release-025deg_jra55_ryf, release-025deg_jra55_iaf_bgc and release-025deg_jra55_ryf_bgc are all at version 2.0 now :)

I think this is done!

[aidanheerdegen]

Great work, thanks @anton-seaice