Unified Forecast System (UFS)
Want to build UFS on Linux-based VM, hosted on Mac laptop (intel), but user's guide is slightly confusing #27
-
|
I want to build UFS - and it seems my platform is a type 3 (system - see the title). However, I'm having trouble following the user's guide. devbuild.sh seemed to not be the way to go. Should I be using the cmake approach? I guess I'm wondering if there is a "one-page" of instructions that could help me build/run the UFS SRW app. I did get the manage_externals/checkout_externals as well as the building blocks in place - gcc/gfortran/perl/etc. done. But could definitely use some pointers on the rest, i.e., HPC-stack install, then SRW app. It seems WRF was easier to set up, but anyway, thanks so much and apologies for such a basic question. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments
-
|
@githubweather - building the UFS SRW App requires software libraries to be installed first. Quick Start Guide Chapter 2 of the SRW Users' Guide is a good starting point, and it refers to building HPC-stack first, eventually pointing to the HPC-stack documentation and building it on a Mac (Chapter 3) As to WRF and UFS SRW: Unlike WRF, SRW App is being developed as end-to-end system, which takes the user from choosing model grid (customized or options for pre-defined) to getting post-processed model data and options to create graphical output. UFS Weather Model is used operationally, which places certain requirements and conventions required for the NOAA operational components. Thus it is indeed essential to get feedback from users on what could be changed to make it more accessible for the community. |
Beta Was this translation helpful? Give feedback.
-
|
@NataliePerlin - thanks for your reply. Just for clarification, I
am actually running in a Debian Linux environment (by way of a virtual
machine on my Macbook). Sorry it has taken a second to get back. I pulled
hpc-stack off github and built it via build_stack.sh. It finished, but I
saw 'Error' listed in the NCO, CDO, wgrib2, madis, and PIO sections.
However, at the end, it said successful install. So I'll assume that is
right and those 'errors' aren't showstoppers.
Went on and:
(1) edited modulefiles/build_linux_gnu.lua, so it had the correct path to
my hpc-stack install
(2) ran "devbuild.sh --platform=linux --compiler=gnu"
this returns:
======================
PLATFORM(MACHINE)=linux
COMPILER=gnu
MODULE_FILE=build_linux_gnu
... Load MODULE_FILE and create BUILD directory ...
*Lmod has detected the following error: * The load_any function failed
because it could not
find any of the following modules : png/1.6.35 libpng/1.6.37
Please check the spelling or version number. Also try "module spider ..."
Also make sure that all modulefiles written in TCL start with the string
#%Module
While processing the following module(s):
Module fullname Module Filename
…--------------- ---------------
srw_common /home/me/ufs-srweather-app/modulefiles/srw_common.lua
build_linux_gnu
/home/me/ufs-srweather-app/modulefiles/build_linux_gnu.lua
=========================
At this point, having never used conda before, I went ahead and installed
anaconda (thought it was safer to have all the capabilities vs. the
lightweight miniconda, although if you advise that miniconda is adequate, I
will switch over, because anaconda eats up a fair amount of space)
Started playing with conda some more to try and figure out how to satisfy
the 'png' errors above. For ex.:
conda config --add channels conda-forge
conda update --all --yes
conda install libpng (but it is saying this package is already installed,
at which point I have hit a wall. Maybe it is just a libpng version number
issue? If so, can you do 'conda install libpng-(some version #)?)
Thanks for any ideas. Excited about getting this to work :>
--Steve
On Sat, Sep 9, 2023 at 12:22 AM Natalie Perlin ***@***.***> wrote:
@githubweather <https://github.com/githubweather> - building the UFS SRW
App requires software libraries to be installed first. Quick Start Guide
Chapter 2 of the SRW Users' Guide
<https://ufs-srweather-app.readthedocs.io/en/develop/BuildingRunningTesting/Quickstart.html#quick-start-guide>
is a good starting point, and it refers to building HPC-stack first,
eventually pointing to the HPC-stack documentation
<https://hpc-stack.readthedocs.io/en/latest/> and building it on a Mac (Chapter
3 <https://hpc-stack.readthedocs.io/en/latest/mac-install.html>)
Building the stack on a Mac OS system is a step that differs from other
systems, but once software libraries are installed, building UFS SRW App
could be done with more "one-page" instructions.
Please let us know about your progress with the build (some module
versions pending updates).
As to WRF and UFS SRW: Unlike WRF, SRW App is being developed as
end-to-end system, which takes the user from choosing model grid
(customized or options for pre-defined) to getting post-processed model
data and options to create graphical output. UFS Weather Model is used
operationally, which places certain requirements and conventions required
for the NOAA operational components. Thus it is indeed essential to get
feedback from users on what could be changed to make it more accessible for
the community.
—
Reply to this email directly, view it on GitHub
<#27 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AINWLDGFCA2Z3JZMNUTRKKDXZQDLPANCNFSM6AAAAAA4PP2R2Y>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
-
|
Forgot to add, also tried these things --
%conda activate workflow_tools
and it returns saying there is no environment 'workflow_tools'.
Then tried:
%conda install workflow_tools
and it thinks for a couple of minutes, then dies saying "Killed".
Interestingly,
%conda install workflows
works just fine.
…--Steve
On Wed, Sep 13, 2023 at 3:18 PM Stephen Kirby ***@***.***> wrote:
@NataliePerlin - thanks for your reply. Just for clarification, I
am actually running in a Debian Linux environment (by way of a virtual
machine on my Macbook). Sorry it has taken a second to get back. I pulled
hpc-stack off github and built it via build_stack.sh. It finished, but I
saw 'Error' listed in the NCO, CDO, wgrib2, madis, and PIO sections.
However, at the end, it said successful install. So I'll assume that is
right and those 'errors' aren't showstoppers.
Went on and:
(1) edited modulefiles/build_linux_gnu.lua, so it had the correct path to
my hpc-stack install
(2) ran "devbuild.sh --platform=linux --compiler=gnu"
this returns:
======================
PLATFORM(MACHINE)=linux
COMPILER=gnu
MODULE_FILE=build_linux_gnu
... Load MODULE_FILE and create BUILD directory ...
*Lmod has detected the following error: * The load_any function failed
because it could not
find any of the following modules : png/1.6.35 libpng/1.6.37
Please check the spelling or version number. Also try "module spider ..."
Also make sure that all modulefiles written in TCL start with the string
#%Module
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
srw_common /home/me/ufs-srweather-app/modulefiles/srw_common.lua
build_linux_gnu
/home/me/ufs-srweather-app/modulefiles/build_linux_gnu.lua
=========================
At this point, having never used conda before, I went ahead and installed
anaconda (thought it was safer to have all the capabilities vs. the
lightweight miniconda, although if you advise that miniconda is adequate, I
will switch over, because anaconda eats up a fair amount of space)
Started playing with conda some more to try and figure out how to satisfy
the 'png' errors above. For ex.:
conda config --add channels conda-forge
conda update --all --yes
conda install libpng (but it is saying this package is already installed,
at which point I have hit a wall. Maybe it is just a libpng version number
issue? If so, can you do 'conda install libpng-(some version #)?)
Thanks for any ideas. Excited about getting this to work :>
--Steve
On Sat, Sep 9, 2023 at 12:22 AM Natalie Perlin ***@***.***>
wrote:
> @githubweather <https://github.com/githubweather> - building the UFS SRW
> App requires software libraries to be installed first. Quick Start Guide
> Chapter 2 of the SRW Users' Guide
> <https://ufs-srweather-app.readthedocs.io/en/develop/BuildingRunningTesting/Quickstart.html#quick-start-guide>
> is a good starting point, and it refers to building HPC-stack first,
> eventually pointing to the HPC-stack documentation
> <https://hpc-stack.readthedocs.io/en/latest/> and building it on a Mac (Chapter
> 3 <https://hpc-stack.readthedocs.io/en/latest/mac-install.html>)
> Building the stack on a Mac OS system is a step that differs from other
> systems, but once software libraries are installed, building UFS SRW App
> could be done with more "one-page" instructions.
> Please let us know about your progress with the build (some module
> versions pending updates).
>
> As to WRF and UFS SRW: Unlike WRF, SRW App is being developed as
> end-to-end system, which takes the user from choosing model grid
> (customized or options for pre-defined) to getting post-processed model
> data and options to create graphical output. UFS Weather Model is used
> operationally, which places certain requirements and conventions required
> for the NOAA operational components. Thus it is indeed essential to get
> feedback from users on what could be changed to make it more accessible for
> the community.
>
> —
> Reply to this email directly, view it on GitHub
> <#27 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AINWLDGFCA2Z3JZMNUTRKKDXZQDLPANCNFSM6AAAAAA4PP2R2Y>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Beta Was this translation helpful? Give feedback.
-
|
Building a software stack is a separate task that need to be done before any SRW installation. If you used HPC-stack, it is not being supported during the past year, and all of the efforts are made to transition to spack-stack (see Documentation). It has a section with guidance on building software stack on new sites that use Linux or MacOS. During the HPC-stack installation you may not get errors when some libraries fail to build. A file hpc-stack-contents.log contains a list of libraries that are installed. Looking through the installation log will help to determine what libraries failed. A directory ./log contains individual installation logs for the packages that could be examined for installation errors. For Linux, a faster way could be using a container approach (https://ufs-srweather-app.readthedocs.io/en/develop/BuildingRunningTesting/ContainerQuickstart.html), when most of the software is pre-packaged. This could be the fastest way. It does require a few additional items installed, such as the following:
Note that |
Beta Was this translation helpful? Give feedback.
-
|
NataliePerlin -
Thx for that excellent update. I'll now move over to spack-stack and work
on configuring that. Greatly appreciate all the links you have provided.
…On Wed, Sep 13, 2023 at 11:25 PM Natalie Perlin ***@***.***> wrote:
@githubweather <https://github.com/githubweather> -
Building a software stack is a separate task that need to be done before
any SRW installation. If you used HPC-stack, it is not being supported
during the past year, and all of the efforts are made to transition to
spack-stack (see Documentation
<https://spack-stack.readthedocs.io/en/latest/index.html>). It has a
section with guidance on building software stack on new sites that use
Linux or MacOS.
During the HPC-stack installation you may not get errors when some
libraries fail to build. A file hpc-stack-contents.log contains a list of
libraries that are installed. Looking through the installation log will
help to determine what libraries failed. A directory ./log contains
individual installation logs for the packages that could be examined for
installation errors.
For Linux, a faster way could be using a container approach (
https://ufs-srweather-app.readthedocs.io/en/develop/BuildingRunningTesting/ContainerQuickstart.html),
when most of the software is pre-packaged. This could be the fastest way.
It does require a few additional items installed, such as the following:
- recent Intel compilers and MPI,
- container software, Singularity/Apptainer
<https://apptainer.org/docs/user/1.2/introduction.html>;
- a workflow manager such as rocoto
<https://christopherwharrop.github.io/rocoto/>, SRW chapter on Rocoto
<https://ufs-srweather-app.readthedocs.io/en/develop/Reference/RocotoInfo.html>.
It is possible to run the SRW without rocoto but is not recommended.
- customized conda environments (see SRW Docs on conda workflow
environment
<https://ufs-srweather-app.readthedocs.io/en/latest/BuildingRunningTesting/RunSRW.html#load-the-conda-python-environment>
)
Note that workflow_tools is not a part of standard conda packages, and
cannot be installed using conda install.
—
Reply to this email directly, view it on GitHub
<#27 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AINWLDBWDR2WAC35WLTYFG3X2KILFANCNFSM6AAAAAA4PP2R2Y>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Beta Was this translation helpful? Give feedback.
@githubweather -
Building a software stack is a separate task that need to be done before any SRW installation. If you used HPC-stack, it is not being supported during the past year, and all of the efforts are made to transition to spack-stack (see Documentation). It has a section with guidance on building software stack on new sites that use Linux or MacOS.
During the HPC-stack installation you may not get errors when some libraries fail to build. A file hpc-stack-contents.log contains a list of libraries that are installed. Looking through the installation log will help to determine what libraries failed. A directory ./log contains individual installation logs for the packages that coul…