[develop] update Gaea modulefile

[natalie-perlin]

DESCRIPTION OF CHANGES:

• Updated build_gaea_intel.lua modulefile to use the new stack location on C3/C4 built following an upgrade with intel-classic-2022.0.2 compiler and cray-mpich/7.7.20.
  File changed:
  ./modulefiles/build_gaea_intel.lua
./modulefiles/wflow_gaea.lua
./modulefiles/tasks/gaea/python_srw.lua
./modulefiles/tasks/gaea/plot_allvars.local.lua
./modulefiles/tasks/gaea/run_vx.local.lua
./ush/machine/gaea.yaml
./lmod-setup.csh

Type of change

• Bug fix (non-breaking change which fixes an issue)

TESTS CONDUCTED:

• Regression tests run for the UFS-WM with the updated stack, all pass successfully as reported in Gaea software stack location update after C3 and C4 partition upgrades ufs-weather-model#1758 . Updated UFS-WM gaea modules merged via PR: Update CMEPS; add ATM ORT test (was #1747); update Gaea stack location (was #1758) ufs-weather-model#1752 (comment)
• Built SRW on Gaea, ran fundamental tests suite. Most of the tests and tasks finished successfully, except for the following:
  -- grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot, task plot_allvars (DEAD)
  -- test nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16, tasks make_ics_mem000, make_lbcs_mem000 (DEAD), and subsequent tasks that were not run
  -- grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16, tasks run_MET_EnsembleStat_vx_APCP01h, run_MET_EnsembleStat_vx_APCP03h , run_MET_EnsembleStat_vx_APCP06h, run_MET_EnsembleStat_vx_REFC, run_MET_EnsembleStat_vx_RETOP, run_MET_EnsembleStat_vx_SFC , run_MET_EnsembleStat_vx_UPA (DEAD)

• gaea.intel
• fundamental test suite

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

[MichaelLueken]

@natalie-perlin I was able to clone your forked branch and successfully build the SRW App using your updated Gaea modulefile. However, I'm noting that there is an issue with the workflow_tools conda environment on Gaea. In PR #793, the regional_workflow conda environment was replaced with workflow_tools. However, on Gaea, workflow_tools isn't currently setup to work with the SRW (while attempting to run either the fundamental or coverage tests, this conda environment is unable to find jinja2). During the SRW CM meeting, you noted that there were changes that needed to be brought over to Gaea following the library changes to the machine. Would the workflow_tools conda environment be one of the modifications that still needs to be addressed before running the SRW will be possible on Gaea?

[natalie-perlin]

@MichaelLueken -
Fundamental tests results attached. Some tests and tasks are reported as "COMLETE", others are "UNAVAILABLE", and one "DEAD". I'm looking into the details of these results. It might be that some scripts for gaea missing, since it was out of use for some time.
WE2E_tests_20230615194157.yaml.txt

[MichaelLueken]

@natalie-perlin I'm seeing similar to what you are, tests and tasks are reporting back with "SUCCEEDED", "UNAVAILABLE", or "DEAD". Interestingly, for the tasks that return "UNAVAILABLE", the log files show that the test successfully ran to completion. It's not clear to me why several tasks are returning "UNAVAILABLE". I am noticing that the "UNAVAILABLE" entries correspond with "STALLED" in the verbose output from ./run_WE2E_tests.py.

I have noted that your branch is seven commits behind the current HEAD of the authoritative develop branch. It might be interesting to see if things change after updating to the latest develop.

[natalie-perlin]

@MichaelLueken - thank you for your comments and looking into the logs!

Launching the WE2E tests by hand may involve explicit steps to initialize Lmod (source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh), to load the module wflow_gaea, and to activate regional_workflow environment. This is now needed because the launch test switch to python. These steps of Lmod and module loading/acitvation are likely hard-coded in Jenkins tests.

[MichaelLueken]

@natalie-perlin You are correct. The Jenkins scripts will force etc/lmod-setup.sh to be used before loading any modulefiles, which does use source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh.

I use csh on the RDHPCS machines, and I'm finding issues with the current etc/lmod-setup.csh script. First, it fails because it is attempting to load modules/3.2.11.4 following the module purge. It looks like there is now a modules/3.2.6.7 module that will likely need to be loaded. Having said that, I have noticed that after using etc/lmod-setup.csh, after adding in modules/3.2.6.7, no modules are being loaded following the module purge. I think that this might be why, when I attempt to run the WE2E tests manually, they fail for me (there is an issue with the etc/lmod-setup.csh script and the necessary modules aren't being loaded before loading wflow_gaea and regional_workflow). Since the Jenkins-based scripts are in bash, they use etc/lmod-setup.sh, which works correctly, which is why I'm able to run WE2E tests that way.

None of this explains why the tests and tasks are returning with "UNAVAILABLE" or "DEAD", unfortunately.

I also need to point out that your branch is missing the changes from PR #793. This PR removed the use of regional_workflow and replaced it with workflow_tools. We need to make sure that workflow_tools is also working correctly before we can move forward with this PR.

[natalie-perlin]

Oh, thanks for pointing out at changes due for the etc/lmod-setup.csh script.
It will indeed need testing with the updates and changes from PR #793.

[natalie-perlin]

Still sorting out workflow issues for Gaea:
where the actually submitted job script for a workflow task could be found?

The whole job script needs to be submitted to a specific cluster (--clusters=c3,c4), not only srun command. What job template would need to be examined or/and possibly updated?

[MichaelLueken]

@natalie-perlin It looks like the jobs are being submitted via the parm/wflow/*.yaml files. Changing SCHED_NATIVE_CMD and SCHED_NATIVE_CMD_HPSS in ush/machine/gaea.yaml should allow you to submit the jobs using the desired clusters. If you have already tried modifying these two settings in Gaea's machine file, then I'm not entirely sure where else the jobs might be submitted.

[natalie-perlin]

@MichaelLueken - thank you, Michael.
I was looking for a final version of a batch script, something like job_card in Weather Model, or fv3_slurm.IN.
The SRW tasks do run successfully to completion on c4, but I'm concerned about some pmi/mpi messages in the output related to alps modulefile for cray, and want to make sure there will be no any cray-related issues down the road.

[natalie-perlin]

All worked OK after removing the following from my previously tested configuration:
PARTITION_DEFAULT: eslogin
PARTITION_FCST: c4

The test nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_GFS_v16 fully passes now:

        CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202208101200           get_extrn_ics                    76034359           SUCCEEDED                   0         1         398.0
202208101200          get_extrn_lbcs                    76034360           SUCCEEDED                   0         1        1444.0
202208101200         make_ics_mem000                   269317853           SUCCEEDED                   0         1         128.0
202208101200        make_lbcs_mem000                   269317854           SUCCEEDED                   0         1         202.0
202208101200         run_fcst_mem000                   269317855           SUCCEEDED                   0         1         808.0
202208101200    run_post_mem000_f000                   269317856           SUCCEEDED                   0         1          31.0
202208101200    run_post_mem000_f001                   269317858           SUCCEEDED                   0         1          25.0
202208101200    run_post_mem000_f002                   269317862           SUCCEEDED                   0         1          25.0
202208101200    run_post_mem000_f003                   269317863           SUCCEEDED                   0         1          24.0
202208101200    run_post_mem000_f004                   269317865           SUCCEEDED                   0         1          25.0
202208101200    run_post_mem000_f005                   269317866           SUCCEEDED                   0         1          25.0
202208101200    run_post_mem000_f006                   269317867           SUCCEEDED                   0         1          24.0

[natalie-perlin]

Branch natalie-perlin:update-gaea-stack is updated as well.

[MichaelLueken]

The fundamental tests have successfully run through to completion:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              11.80
nco_grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_timeoffset_suite_  COMPLETE              15.18
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               9.63
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              18.89
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              31.09
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_RAP_suite_WoFS_v0              COMPLETE              15.15
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              26.08
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             127.82

Will now try running the coverage tests using the Jenkins scripts.

[MichaelLueken]

The Gaea WE2E coverage test suite successfully ran through to completion:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community                                                          COMPLETE              22.37
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE              28.93
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP              COMPLETE              30.82
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              33.32
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson  COMPLETE             366.97
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR          COMPLETE              30.55
grid_RRFS_CONUScompact_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta     COMPLETE             359.47
grid_SUBCONUS_Ind_3km_ics_RAP_lbcs_RAP_suite_RRFS_v1beta_plot      COMPLETE              10.02
nco_ensemble                                                       COMPLETE              81.18
nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thom  COMPLETE             361.95
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            1325.58

We should re-add Gaea to the Jenkinsfile so that the automated tests will once again run on Gaea. How would you like to go about doing this? I can open a PR in your fork to update the .cicd/Jenkinsfile, you can update the .cicd/Jenkinsfile, or I can update the .cicd/Jenkinsfile in PR #859 and have you merge in PR #859 before submitting tests. With the resource issue on Jet (hfv3gfs quota has been exceeded, so moving to Jet-EPIC instead), this third approach might be the best way. In the interim, I will go ahead and approve this PR, since the SRW both builds and runs on Gaea now.

[MichaelLueken]

@natalie-perlin -

The SRW App both builds and runs (both fundamental and coverage tests have been successfully run) on Gaea, I will now approve these changes.

We will need to decide how to proceed with updating the Jenkinsfile (reactivating Gaea and moving from Jet to Jet-EPIC since Jet's hfv3gfs account's quota has been exceeded) before adding the Jenkins label to this PR.

[MichaelLueken]

@natalie-perlin -

@BruceKropp-Raytheon's Functional Workflow Task Tests have failed on Gaea (.cicd/scripts/srw_ftest.sh). It is failing due to being unable to load the miniconda3/4.12.0 module because the python module is already loaded. I'll dig into this more and see if I can't find a fix that will work for Gaea.

[natalie-perlin]

@MichaelLueken - please let me know if I could help and look into the logs

[MichaelLueken]

The directory that contains the experiment on Gaea is - /lustre/f2/dev/wpo/role.epic/jenkins/workspace/fs-srweather-app_pipeline_PR-836

and the pipeline can be viewed - https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-836/1/pipeline/232

[MichaelLueken]

@natalie-perlin -

I suspect that the issue is that a version of python is loaded by default when using:

source /lustre/f2/dev/role.epic/contrib/Lmod_init.sh

and then modulefiles/wflow_gaea.lua is attempting to load miniconda3/4.12.0, leading to the failure. I'll try adding a module unload python in .cicd/scripts/srw_ftest.sh and see if that corrects the behavior. Otherwise, it might require adding a module unload python in the modulefiles/wflow_gaea.lua file.

[MichaelLueken]

I was able to find a fix and have pushed it. The fix was also tested on Hera. Will resubmit the Jenkins tests on Gaea.

[natalie-perlin]

@MichaelLueken -
Awesome, thanks. I figured out there is still an issue for me to access Jenkins.

[natalie-perlin]

@MichaelLueken -
I found indeed something not consistent. On the login node, say, gaea10, gaea12 or gaea13 (and others, as far as I remember), no python module is loaded by default. E.g.,

[Natalie.Perlin@Gaea:~]$ hostname
gaea10
[Natalie.Perlin@Gaea:~]$ module list

Currently Loaded Modules:
  1) craype/2.7.15                13) craype-network-aries
  2) intel-classic/2022.2.1       14) rca/2.2.22-7.0.4.1_2.35__ged51428.ari
  3) pmi/5.0.17                   15) udreg/2.3.2-7.0.4.1_2.33__g5f0d670.ari
  4) cray-libsci/22.05.1          16) ugni/6.0.14.0-7.0.4.1_3.35__ge0d449e.ari
  5) perftools-base/23.02.0       17) dmapp/7.1.1-7.0.4.1_2.37__gcec52bc.ari
  6) PrgEnv-intel/6.0.10-classic  18) gni-headers/5.0.12.0-7.0.4.1_2.33__gd0d73fe.ari
  7) craype-broadwell             19) xpmem/2.2.29-7.0.4.1_2.25__g35859a4.ari
  8) CmrsEnv                      20) job/2.2.5-7.0.4.1_2.41__gcc91aa9.ari
  9) TimeZoneEDT                  21) dvs/2.15_2.2.244-7.0.5.0_52.76__g6842f22a
 10) globus-toolkit/6.0.17        22) alps/6.6.67-7.0.4.1_2.36__gb91cd181.ari
 11) darshan/3.2.1                23) cray-mpich/7.7.20
 12) DefApps                      24) eproxy/2.0.24-7.0.4.1_2.18__g45aade1.ari

In the Jenkins file in your link, python/3.9 is loaded as one of the default modules, which is not expected. Please see lines 354 and 357 of

https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-836/1/pipeline/232/

++++ echo craype/2.7.15:intel-classic/2022.2.1:pmi/5.0.17:cray-libsci/22.05.1:perftools-base/23.02.0:PrgEnv-intel/6.0.10-classic:craype-broadwell:CmrsEnv:TimeZoneEDT:globus-toolkit/6.0.17:darshan/3.2.1:DefApps:python/3.9:craype-network-aries:rca/2.2.22-7.0.4.1_2.35__ged51428.ari:udreg/2.3.2-7.0.4.1_2.33__g5f0d670.ari:ugni/6.0.14.0-7.0.4.1_3.35__ge0d449e.ari:dmapp/7.1.1-7.0.4.1_2.37__gcec52bc.ari:gni-headers/5.0.12.0-7.0.4.1_2.33__gd0d73fe.ari:xpmem/2.2.29-7.0.4.1_2.25__g35859a4.ari:job/2.2.5-7.0.4.1_2.41__gcc91aa9.ari:dvs/2.15_2.2.244-7.0.5.0_52.76__g6842f22a:alps/6.6.67-7.0.4.1_2.36__gb91cd181.ari:cray-mpich/7.7.20:eproxy/2.0.24-7.0.4.1_2.18__g45aade1.ari
+++ loaded_modules='craype/2.7.15
intel-classic/2022.2.1
pmi/5.0.17
cray-libsci/22.05.1
perftools-base/23.02.0
PrgEnv-intel/6.0.10-classic
craype-broadwell
CmrsEnv
TimeZoneEDT
globus-toolkit/6.0.17
darshan/3.2.1
DefApps
python/3.9
craype-network-aries
rca/2.2.22-7.0.4.1_2.35__ged51428.ari
udreg/2.3.2-7.0.4.1_2.33__g5f0d670.ari
ugni/6.0.14.0-7.0.4.1_3.35__ge0d449e.ari
dmapp/7.1.1-7.0.4.1_2.37__gcec52bc.ari
gni-headers/5.0.12.0-7.0.4.1_2.33__gd0d73fe.ari
xpmem/2.2.29-7.0.4.1_2.25__g35859a4.ari
job/2.2.5-7.0.4.1_2.41__gcc91aa9.ari
dvs/2.15_2.2.244-7.0.5.0_52.76__g6842f22a
alps/6.6.67-7.0.4.1_2.36__gb91cd181.ari
cray-mpich/7.7.20
eproxy/2.0.24-7.0.4.1_2.18__g45aade1.ari'

This indeed would create an issue with loading miniconda3. I wonder why this did not cause troubles in my tests (unless they were overlooked, and did not cause any failures!)
So yes, adding "module unload python", or it's equivalent in lua module, should help.

[MichaelLueken]

@natalie-perlin -

It looks like the issue might be in /lustre/f2/dev/role.epic/contrib/Lmod_init.sh. Looking at the output, it looks like python/3.9 is being loaded because it is contained in DefApps.

The fix I pushed on Friday only adds module unload python to the .cicd/scripts/srw_ftest.sh script, not the modulefiles/build_gaea_intel.lua file. If you would like, I can attempt to add the module unload python to the build_gaea_intel.lua modulefile and see if it works, then resubmit the tests for Gaea. Please let me know what you would like to do here.

[natalie-perlin]

@MichaelLueken -
Trying to avoid unloads when not required, and so I've added unloading python module in wflow_gaea.lua modulefile, right before the miniconda3 is loaded that created a conflict in the first place. Please let me know if this works out.

[MichaelLueken]

The Cheyenne Intel tests were manually run on Hera and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
custom_GFDLgrid__GFDLgrid_USE_NUM_CELLS_IN_FILENAMES_eq_FALSE      COMPLETE              11.10
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              30.26
grid_RRFS_CONUS_25km_ics_NAM_lbcs_NAM_suite_GFS_v16                COMPLETE              19.72
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_HRRR           COMPLETE              25.79
grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta    COMPLETE               8.67
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_HRRR                COMPLETE              16.57
pregen_grid_orog_sfc_climo                                         COMPLETE               7.71
specify_template_filenames                                         COMPLETE               7.49
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             127.31

The Cheyenne GNU tests were manually run on Hera and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_CONUS_25km_GFDLgrid_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16      COMPLETE              20.68
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta      COMPLETE             236.32
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_2017_gfdlmp  COMPLETE             110.21
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v17_p8_plot  COMPLETE              28.16
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR             COMPLETE              37.13
grid_RRFS_CONUScompact_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16   COMPLETE              23.19
grid_RRFS_NA_13km_ics_FV3GFS_lbcs_FV3GFS_suite_RAP                 COMPLETE             319.38
grid_SUBCONUS_Ind_3km_ics_NAM_lbcs_NAM_suite_GFS_v16               COMPLETE              51.47
specify_EXTRN_MDL_SYSBASEDIR_ICS_LBCS                              COMPLETE              10.40
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             836.94

The Jet tests were manually run and all successfully passed:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
community                                                          COMPLETE              17.29
custom_ESGgrid                                                     COMPLETE              15.30
custom_GFDLgrid                                                    COMPLETE              11.05
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_nemsio_2021032018         COMPLETE               8.93
get_from_HPSS_ics_FV3GFS_lbcs_FV3GFS_fmt_netcdf_2022060112_48h     COMPLETE              50.08
get_from_HPSS_ics_RAP_lbcs_RAP                                     COMPLETE              16.07
grid_RRFS_AK_3km_ics_FV3GFS_lbcs_FV3GFS_suite_HRRR                 COMPLETE             232.91
grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16_plot     COMPLETE              38.10
grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2        COMPLETE               7.73
grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_RRFS_v1beta       COMPLETE             509.80
nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR       COMPLETE              11.29
process_obs                                                        COMPLETE               0.25
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             918.80

The rerun of the automated tests on Gaea also succeeded.

Moving forward with merging this work now.