Model run for few minutes and exited automatically

[ctensubam]

Hi everyone,
When I run ACCESS-OM2 with my configuration (including surface wave mixing) the model run for few minutes like 5 mins and exited automatically throwing this error:
payu: Model exited with error code 174; aborting.

And I find the followings in the error log

Regarding my input data, I have changed variable attributes as required and also changed all the missing values from nans to 0 where the land masked are.
Kindly help how to fix this!

[aekiss]

Hi Chinglen,

How are you building access-om2? Does it work for you if you compile the version in https://github.com/COSIMA/access-om2 master?

The master branch is actually quite out of date, so that could also be the problem. I'll update it when we finalise some changes to the model components.

Re. input files - we normally fill in the land mask nans with interpolations (instead of zero) so the model gets reasonable data if there is a land mask misalignment - e.g. see https://github.com/COSIMA/initial_conditions_access-om2 and its various submodules.

[ctensubam]

Hi Andrew,
Yes, first I downloaded the repository from https://github.com/COSIMA/access-om2 and then , I changed three fortran source codes namely ocean_vert_gotm.F90, turbulence.F90 and production.F90. After that, I recompiled the model and created this ocean executable /g/data/qw43/ct2143/ACCESS-OM2/access-om2/bin/fms_ACCESS-OM-BGC_af3a94d-modified_libaccessom2_a227a61.x

And, I also changed some module namelist switches in /ocean/input.nml to use k-epsilon mixing scheme instead of kpp model (MOM's default)

And I passed my wave input in /ocean/data_table like

The model run completely fine if I don't switch on wave mixing as described above. The problem arises when I include wave
mixing that the model runs for few minutes and exited automatically.
Regarding my input data, before I changed fill values to zeros, I had also tried with nans but didn't work.

I was wondering if it is the timestep and start time used in accessom2.nml.
The default time step is 1350s. And, I set the model start time from 2015-02-01T00:00:00 and my input data start from 2015-01-01T01:30:00 (keeping input timestep same as JRA-55 forcing used in the model). I made the model to start one month after my input start time because if I make the model and input start time be same, it throws an error.

Regards
Chinglen

[ctensubam]

Thanks Andrew,
I have fixed the problem. The model now runs fine with wave input setting.

And, I need some clarification regarding ACCESS-OM25+BGC that is :
When I run 025deg_jra55_iaf master+bgc configuration https://github.com/COSIMA/025deg_jra55_iaf/tree/master+bgc using my ocean executable fms_ACCESS-OM-BGC_af3a94d-modified_libaccessom2_a227a61.x it throws an error saying

How can I fix this error?
This error doesn't come up when I use the default ocean executable. I used the same INPUT paths as given in config.yaml file
but can't find these three files 'ocmip2_press_monthly_om2.nc', 'ocmip2_xkw_monthly_om2.nc', and 'ocmip2_fice_monthly_om1p5_bc.nc' in any INPUT paths given in config file.

[aekiss]

Hi Chinglen, sorry for my slow reply. Have you been able to fix this new problem?

[ctensubam]

Hi Andrew,
I have fixed the previous issue. But the model keeps on crashing with BGC configuration. It works fine without BGC. I solved the similar issue by reducing time step from 5400 to 2700 in configuration (1deg_iaf) without BGC. But it keeps on crashing with BGC even I reduce/change timestep. The error says:

forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
fms_ACCESS-OM-BGC 0000000001C84744 Unknown Unknown Unknown
libpthread-2.28.s 000014EF1751ECE0 Unknown Unknown Unknown

Regards
Chinglen

[aekiss]

Hi Chinglen, that seems odd but the error message you've posted doesn't provide enough information for me to diagnose the problem.

Do you need BGC?

[ctensubam]

Hi Andrew,
The error file is attached.
Yes, I need BGC. I downloaded ACCESS-OM2-BGC configuration from master+bgc branch https://github.com/COSIMA/1deg_jra55_iaf.git
and ran the model using my executables. I am also attaching input.nml and field_table for my experiment, if needed.

Regards
Chinglen

access-om2.err.txt

field_table.txt
input.nml.txt

[aekiss]

The problem is ice: bad initialization array name nit_io

Looks like you are missing an initial condition for ice nitrate. Are you using the same inputs in config.yaml as here?
https://github.com/COSIMA/1deg_jra55_iaf/blob/master%2Bbgc/config.yaml

Also, if you only want ocean BGC you can turn off ice BGC - see README.

[ctensubam]

Hi Andrew,
Yes, I used the same input paths in config.yaml file. Here I am attaching my config.yml file.
config.yaml.txt

I think all the required IC files for BGC are not present in the input paths mentioned in config.yaml file. For example, previously, I couldn't find these three files 'ocmip2_press_monthly_om2.nc', 'ocmip2_xkw_monthly_om2.nc', and 'ocmip2_fice_monthly_om1p5_bc.nc' in the input paths of config.yaml. So, I downloaded 'p','xkw' and'fice' monthly data from OCMIP2 website (http://ocmip5.ipsl.jussieu.fr/OCMIP/phase2/simulations/) and used for my experiment.

Now, I turn off ice BGC and run again. Now, the error is different.
FATAL from PE 170: check_nml_error in fms_mod: Unknown error while reading namelist mom_oasis3_interface_nml, (IOSTAT = 18 )

The error file is here
access-om2.51757236.gadi-pbs.err.txt

Thanks
Chinglen

[aekiss]

I'm pretty confused by all this. Which model component needs 'ocmip2_press_monthly_om2.nc', 'ocmip2_xkw_monthly_om2.nc', and 'ocmip2_fice_monthly_om1p5_bc.nc'? They're not needed by MOM or CICE in the usual ACCESS-OM2 configs. Are they for WW3?

Regarding your new error, did you change something in mom_oasis3_interface_nml?

[ctensubam]

Hi Andrew,
It is used in MOM5. Please check in ocean/field_table (https://github.com/COSIMA/1deg_jra55_iaf/blob/master%2Bbgc/ocean/field_table) like below:

"namelists","ocean_mod","csiro_bgc/global"
use_access_co2 = .false.
gasx_from_file = .false.
ice_file4gasx = .false.
zero_floor = .true.
atmpress_file = INPUT/ocmip2_press_monthly_om2.nc
atmpress_name = p
pistonveloc_file = INPUT/ocmip2_xkw_monthly_om2.nc
pistonveloc_name = xkw
seaicefract_file = INPUT/ocmip2_fice_monthly_om1p5_bc.nc
seaicefract_name = fice

And, the model throws errors if above three files are not present in the input paths. But I have already prepared these files.

Regarding new error, it is due to additional 3 field_in 'wnd_io', 'iof_nit', 'iof_alg' and 2 additional field_out 'n_surf', 'alg_surf' in the ocean/input.nml file in BGC configuration. These additional fields are not used in master branch.

The model will throw 'ice: bad initialization array name nit_io' error if I removed these additional field_in and field_out.

Regards
Chinglen

[aekiss]

Those 3 files in ocean/field_table should ignored if you run from the master+bgc branch of a fresh clone of https://github.com/COSIMA/1deg_jra55_iaf, so you shouldn't need to provide them.
Is this what you find? If so, there must be something different about the configuration or executables you're using.

Re. the new error, these 3 fields shouldn't be used the master branch. Are you trying to run from a restart generated with BGC? You may need to edit the restart files to get that to work.

[aekiss]

Did you base your code on the latest MOM5 from the master branch at https://github.com/mom-ocean/MOM5?

[ctensubam]

Hi Andrew,
I tried once again from the beginning. I recompiled and generated the executables : /g/data/qw43/ct2143/access-om2/bin/fms_ACCESS-OM-BGC_af3a94d-modified_libaccessom2_a227a61.x
And, I did git clone -b master+bgc https://github.com/COSIMA/1deg_jra55_iaf
Then, I update config.yaml file to reflect my executables and my project. And, I also turned off ice bgc by setting skl_bgc = .false. in ice/cice_in.nml. But the model throws the same previous error asking for those three files:

FATAL from PE 46: MPP_OPEN:INPUT/ocmip2_press_monthly_om2.nc does not exist.

Image PC Routine Line Source
fms_ACCESS-OM-BGC 0000000001C7669A Unknown Unknown Unknown
fms_ACCESS-OM-BGC 00000000017CC56C mpp_mod_mp_mpp_er 58 mpp_util_mpi.inc
fms_ACCESS-OM-BGC 00000000017EA589 mpp_io_mod_mp_mpp 599 mpp_io_connect.inc
fms_ACCESS-OM-BGC 00000000017DAE0F time_interp_exter 282 time_interp_external.F90

The problem is only when I run from BGC configuration. The model runs fine without BGC.

Regarding restart, for now, I am not using any restart yet. I will use restarts from previous BGC experiments for longer run later on.

[aekiss]

I can't see /g/data/qw43/ct2143/access-om2 because I'm not in qw43.
It seems there's something about your MOM executable that's causing this issue.
I suggest you do

cd /g/data/qw43/ct2143/access-om2
git fetch
gitk --all &

to see if your MOM source includes the latest updates in origin/master.

[ctensubam]

The codes are the latest ones. I usually do git pull before installation. Also, I check the following
git diff master remotes/origin/master
git diff master+bgc remotes/origin/master+bgc
I find no difference.