Updated input file template docuemntatioan and README.

Set HipFT code version to 1.0.0.

README.md

@@ -1,66 +1,120 @@
-![HipFT](hipft_logo.png)
-
+<img width=500 src="hipft_logo.png" alt="HipFT" />
 # HipFT: High-performance Flux Transport 
-Predictive Science Inc.  
-www.predsci.com
-
+[Predictive Science Inc.](https://www.predsci.com)  
+
+--------------------------------  
+
 ## OVERVIEW ##
+<img align="right" src="hipft_example.gif" alt="HipFT Example"/>
+HipFT is a Fortran 2023 code that is used as the computational core of the Open-source Flux Transport (OFT) software suite.  OFT is a complete system for generating full-Sun magnetograms through acquiring & processing observational data, generating realistic convective flows, and running the flux transport model.  
 
-HipFT is the computational core of the Open-source Flux Transport (OFT) software suite, which is a data-assimilation flux transport model used to generate an ensemble of synchronic radial magnetic field maps for use as boundary conditions for the coronal field models.   
+HipFT implements advection, diffusion, and data assimilation for the solar surface on a logically rectangular nonuniform spherical grid.  It is written in Fortran and parallelized for use with multi-core CPUs and GPUs using a combination of Fortran's standard parallel `do concurrent` (DC), OpenMP Target data directives and MPI.  It uses high-order numerical methods such as SSPRK(4,3), Strang splitting, WENO3, and the super time-stepping scheme RKG2.  The code is designed to be modular, incorporating various differential rotation, meridianal flow, super granular convective flow, and data assimilation models.  It can also compute multiple realizations in a single run spanning multiple choices of parameters.  
 
-HipFT implements advection, diffusion, and data assimilation for the solar surface on a logically rectangular non-uniform spherical grid.  It is written in Fortran and parallelized for use with multi-core CPUs and GPUs using a combination of Fortran's standard parallel `do concurrent` (DC) and OpenMP Target data directives.  To alleviate the strict time-step stability criteria for the diffusion equation, we use an extended stability Runge-Kutta super time-stepping algorithm.  The code is designed to be modular, incorporating various differential rotation, meridianal flow, super granular convective flow, and data assimilation models.
+HipFT can be used with MACOS, Linux, and Windows (through WSL) on CPUs and GPUs (NVIDIA or Intel).  
 
 --------------------------------  
 
-## HOW TO BUILD HIPFT ##  
-
-Copy a build script from the `build_examples` folder that is closest to your setup to the base directory.  
-Modify the script to set the `HDF5` library paths/flags and compiler flags compatible with your system environment.   
-Then, run the script to build `HIPFT` (for example, `./my_build.sh`).  
+## HOW TO BUILD HIPFT ##
+
+HipFT has been tested to work using GCC's `gfortran` (>8), Intel's `ifx` (>21), or NVIDIA's `nvfortran` (>21.5) compilers.  Note that it is NOT compatible with the older Intel `ifort` compiler.  
+It is recommended to use the latest compiler version available.  
+
+1. Find the build script from the `build_examples` folder that is closest to your setup and copy it into the top-level directory.  
+2. Modify the script to set the `HDF5` library paths/flags and compiler flags compatible with your system environment.  
+3. Modify the script to set the compiler options to reflect your setup.  
+4. If running HipFT on CPUs, set your `OMP_NUM_THREADS` (and `ACC_NUM_CORES` if using `nvfortran`) environment variable to the number of threads per MPI rank you want to run with.  
+5. Run the build script (for example, `./my_build.sh`).  
+6. It is recommended to add the `bin` folder to your system path.  
 
-See the multiple build example scripts in the `build_examples` folder for more details.  
+### RUN THE HIPFT TESTSUITE ###
 
-To use the python post processing scripts, add the `HIPFT` `bin` folder to your `PATH`.  
+To test if the installation is working, we recommend running the testsuite after installation.  
+To do this, enter the `testsuite/` directory and run:  
+
+`./run_test_suite.sh`  
+
+This will run the tests with `bin/hipft` using 1 MPI rank.  
+
+IMPORTANT:  If you are building/running HipFT on a multi-core CPU, you will most likely need to  
+use the `-mpicall` option to the `run_test_suite.sh` script to set the proper thread affinity.  
+For example:  For OpenMPI, one would likely want to use `-mpicall="mpirun --bind-to socket -np"`.
 
 --------------------------------  
 
-## HOW TO USE HIPFT ##  
+## HOW TO RUN HIPFT ##
 
 ### Setting Input Options  
 
-`HIPFT` uses a namelist in an input text file called `hipft.dat`.  
+`HIPFT` uses a namelist in an input text file.  The default name for the input is `hipft.in`  
+
+A full working input file with all the default parameter options is provided in the file:  
+
+`doc/hipft.in.documentation` 
+
+A detailed description of each parameter is also given in that file, and (in addition to this readme) is the current main documentation of the code.  
+
+We have also provided example input files for use cases in the `examples/` folder as well as in the testsuite.  
 
-### Launching the Code ###  
+### Launching the Code ###
 
 To run `HIPFT`, set the desired run parameters in a file (e.g. `hipft.in`), then copy or link the `hipft` executable into the same directory as the input file and run the command:  
-  `<MPI_LAUNCHER> -np <N> ./hipft <input_file>`  
-where `<N>` is the total number of MPI ranks to use and `<MPI_LAUNCHER>` is your MPI run command (e.g. `mpiexec`,`mpirun`, `ibrun`, `srun`, etc).  
-For example:  `mpiexec -np 4 ./hipft hipft.in`  
+
+`<MPI_LAUNCHER> <MPI_OPTIONS> -np <N> ./hipft <input_file>`  
+
+where `<N>` is the total number of MPI ranks to use, `<MPI_LAUNCHER>` is your MPI run command (e.g. `mpiexec`,`mpirun`, `ibrun`, `srun`, etc), and `<MPI_OPTIONS>` are additional MPI options that may be needed (such as `--bind-to socket` or `--bind-to numa` for CPUs running with OpenMPI).  
+
+For example:  `mpirun -np 1 ./hipft hipft.in`  
 
 The MPI ranks split up the number of realizations to work on.  
-Therefore, if you are only running 1 realization, you should use 1 MPI rank.  
+Therefore, if you are only running 1 realization, you must use only 1 MPI rank.  
 The number of ranks cannot be larger than the number of realizations.  
 
-The code is parallelized with DC and OpenMP Target data directives across each MPI rank.  
+The code is parallelized with Fortran `do concurrent` and OpenMP Target data directives within each MPI rank.  
+
+### Running HIPFT on CPUs ###
+
+On CPUs, the code is multi-threaded for each MPI rank.  This can require proper setting of the `OMP_NUM_THREADS` and `ACC_NUM_CORES` environment variables (and for GCC, setting them before compilation).  
+It also requires properly setting the thread affinity in the launch of MPI as shown above.  
+For example, running HipFT compiled for GCC and OpenMPI on 4 compute nodes with dual-socket 64-core EPYC CPUs (setup in the BIOS as 4 NUMA domains per socket and no hyperthreading) with more than 16 realizations could be compiled with `OMP_NUM_THREADS=16` and launched with:  
+
+`mpirun --bind-to numa --ntasks-per-node 8 ./hipft hipft.in 1>hipft.log 2<hipft.err`  
+
+A simpler example of running on a single desktop (1 socket), with `OMP_NUM_THREADS` having been set to the total number of threads (before compilation for GCC, at run time with NV and IFX):
+
+`mpirun --bind-to socket -np 1 ./hipft hipft.in 1>hipft.log 2<hipft.err` 
+
+Depending on the system setup, it may be difficult to actualize the full possibly performance on CPU nodes.  We therefore highly recommend running HipFT on GPUs.  
 
 ### Running HIPFT on GPUs ###
 
-For standard cases, one should launch the code such that the number of MPI ranks per node is equal to the number of GPUs per node (assuming you are running at least that many reaslizations). 
+For standard cases, the code should be launched with the number of MPI ranks per node being equal to the number of GPUs per node (assuming you are running at least that many realizations). 
 e.g.  
-`mpiexec -np <N> --ntasks-per-node 4 ./hipft`  
+`mpiexec --ntasks-per-node 4 ./hipft 1>hipft.log 2<hipft.err`  
 or  
-`mpiexec -np <N> --npersocket 2 ./hipft`  
+`mpiexec --npersocket 2 ./hipft 1>hipft.log 2<hipft.err`  
 
-### Solution Output ###  
+Having more data on a GPU typically makes it more efficient.  Therefore for a given number of realizations, it is recommended to try different combinations of numbers of GPUs (e.g. 4 realizations on 2 GPUs (2 per GPU) versus 4 realizations on 1 GPU).  
 
-[This section to be filled in later]  
+### Solution Output ###
 
-### Helpful Scripts ###  
+The output of HipFT are HDF5 map files in phi-theta coordinates.  
+When running with multiple realizations, the output becomes a 3D file with one dimension along realization number.
+In the `bin/` folder, we have provided python scripts for reading in the data and plotting it.
 
-Some useful python scripts for reading, extracting, and plotting the HIPFT input/output data can be found in the  `bin` folder.  
+### Processing Scripts ###
+
+The `/bin` folder contains several python and bash scripts that can be used to post process and plot results of a HipFT run.  Full documentation on their use is pending, however each script has some level of documentation within it.  Check back here for a list of common commands to process the runs.
 
 --------------------------------
 
-### Sample Input Data for Convective Flows and Data Assimilation ###  
+## Sample Data for Convective Flows and Data Assimilation
+
+To use HipFT with data assimilation and convective flows requires that these input data be available.  We plan to release ConFlow and OFTpy which can generate these required data, but for now we have provided sample sets of these data in a zenodo data set located here:  
+
+[HipFT Sample Input Dataset for Convective Flows and Data Assimilation](https://zenodo.org/doi/10.5281/zenodo.10271120)
+
+This data package contains a Carrington rotation of ConFlow convective flows at 15 minute cadence as well as a year of OFTpy HMI-derived data assimilation maps for 2022.  We have also provided an example HipFT input file for running a full year simulation using this data in this repo's `examples/flux_transport_1yr_flowCAa_diff175_data_assim/` folder.  
+Note that the Conflow flow files are auto-repeated in HipFT when run for longer than a Carrington rotation, so they can be used for arbitrary length runs of HipFT.  
 
-https://zenodo.org/doi/10.5281/zenodo.10271120
+--------------------------------
+

bin/hipft_make_plots_and_movies.py

@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+import argparse
+import os
+import h5py
+
+def argParsing():
+  parser = argparse.ArgumentParser(description='HipFt Movie Maker.')
+
+  parser.add_argument('-dir',
+    help='Directory of output files (full path!)',
+    dest='datadir',
+    type=str,
+    required=True)
+
+  parser.add_argument('-dpi',
+    help='DPI for plots.',
+    dest='dpi',
+    default=128,
+    type=int,
+    required=False)
+
+  parser.add_argument('-odir',
+    help='Output directory of movie (Default: current directory).',
+    dest='odir',
+    type=str,
+    required=False)
+
+  parser.add_argument('-o',
+    help='Base filename for output mov movies.',
+    dest='outfile',
+    default='hipft_movie',
+    type=str,
+    required=False)
+
+  parser.add_argument('-slice',
+    help='Slice index (if not specified all slices wil be plotted).',
+    dest='s',
+    type=int,
+    required=False)
+
+  parser.add_argument('-cmin',
+    help='Colormap Minimum',
+    dest='cmin',
+    default='-25',
+    type=float,
+    required=False)
+
+  parser.add_argument('-cmax',
+    help='Colormap Maximum',
+    dest='cmax',
+    default='25',
+    type=float,
+    required=False)
+
+  parser.add_argument('-label',
+    help='Colorbar Label',
+    dest='label',
+    default='Gauss',
+    type=str,
+    required=False)
+
+  parser.add_argument('-mlut',
+    default='hipft_output_map_list_ut.out',
+    required=False,
+    help='Name of the HipFT map text file with ut dates.')
+
+  return parser.parse_args()
+
+
+def run(args):
+  if args.odir:
+    odir=args.odir
+  else:
+    odir=os.getcwd()
+
+  ut_dates=[]
+
+  if os.path.exists(args.mlut):
+    with open(args.mlut, "r") as ftmp:
+      next(ftmp)
+      for line in ftmp:
+        ut_dates.append(line.split()[4])
+  elif os.path.exists('hipft_output_map_list.out'):
+    with open('hipft_output_map_list.out', "r") as ftmp:
+      next(ftmp)
+      for line in ftmp:
+        ut_dates.append(str(float(line.split()[1]))+' hours')
+
+  if not os.path.exists(args.datadir+'/plots'):
+    os.makedirs(args.datadir+'/plots')
+
+  os.chdir(args.datadir+'/plots')
+
+  idx=0
+  for filetmp in os.listdir(args.datadir):
+    file=args.datadir+'/'+filetmp
+    if filetmp.endswith('.h5'):
+      idxx=int(filetmp[-9:-3])
+
+      TITLE=ut_dates[idxx-1]
+      idx+=1
+      with h5py.File(file,'r') as f1:
+        twoD=True
+        if 'dim3' in list(f1.keys()):
+          dim3 = f1['dim3'].shape[0]
+          twoD=False
+
+      if twoD:
+        os.system('plot2d -title "'+ TITLE +'" -cmin '+ str(args.cmin) +' -cmax '+ str(args.cmax) +' -dpi ' + str(args.dpi) \
+          +' -tp -ll -finegrid -unit_label '+ args.label + " "+ file +' -o "'+ os.path.basename(file).replace('.h5','.png'+'"'))
+      elif (args.s == "all"):  
+        for i in range(1,dim3):
+          os.system('hipft_extract_realization.py '+ file +' -r '+ i +' -o tmp_file.h5')
+          fslice="%06d" % (i)
+          os.system('plot2d -title "'+ TITLE +'" -cmin '+ str(args.cmin) +' -cmax '+ str(args.cmax) +' -dpi ' + str(args.dpi) \
+            +' -tp -ll -finegrid -unit_label '+ args.label +' tmp_file.h5 -o "'+ os.path.basename(file).replace('.h5','') +'_r' + str(fslice) +'.png"')
+      elif (args.s):
+        if (args.s > dim3):
+          print("Slice requested is outside range defaulting to last slice : "+ str(dim3))
+          os.system('hipft_extract_realization.py '+ file +' -r '+ str(dim3) +' -o tmp_file.h5')
+          fslice="%06d" % (dim3)
+          os.system('plot2d -title "'+ TITLE +'" -cmin '+ str(args.cmin) +' -cmax '+ str(args.cmax) +' -dpi ' + str(args.dpi) \
+            +' -tp -ll -finegrid -unit_label '+ args.label +' tmp_file.h5 -o "'+ os.path.basename(file).replace('.h5','') +'_r' + str(fslice) +'.png"')
+        else:
+          os.system('hipft_extract_realization.py '+ file +' -r '+ str(args.s) +' -o tmp_file.h5')
+          fslice="%06d" % (args.s)
+          os.system('plot2d -title "'+ TITLE +'" -cmin '+ str(args.cmin) +' -cmax '+ str(args.cmax) +' -dpi ' + str(args.dpi) \
+            +' -tp -ll -finegrid -unit_label '+ args.label +' tmp_file.h5 -o "'+ os.path.basename(file).replace('.h5','') +'_r' + str(fslice) +'.png"')
+      else:
+        os.system('hipft_extract_realization.py '+ file +' -r 1 -o tmp_file.h5')
+        os.system('plot2d -title "'+ TITLE +'" -cmin '+ str(args.cmin) +' -cmax '+ str(args.cmax) +' -dpi ' + str(args.dpi) \
+            +' -tp -ll -finegrid -unit_label '+ args.label +' tmp_file.h5 -o "'+ os.path.basename(file).replace('.h5','') +'_r000001.png"')
+  if os.path.exists("tmp_file.h5"):
+    os.remove("tmp_file.h5")
+
+  if not os.path.exists(args.datadir+'/plots/tmp'):
+    os.makedirs(args.datadir+'/plots/tmp')
+
+  os.chdir(args.datadir+'/plots/tmp')
+
+
+  if twoD:
+    for filetmp in os.listdir(args.datadir+'/plots'):
+      file=args.datadir+'/plots/'+filetmp
+      if filetmp.endswith('.png'):
+        idxx=int(filetmp[-10:-4])
+        os.system('ln -s '+file+' movie'+ str(idxx) +'.png')
+    os.system('ffmpeg -framerate 15 -i "movie%d.png" -pix_fmt yuv420p -c:a copy -crf 20 -r 15 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -codec:v libx264 "movie.mov"')
+    os.rename(args.datadir+'/plots/tmp/movie.mov',odir+'/'+args.outfile+'.mov')
+  elif (args.s == "all"):
+    for j in range(1,dim3):
+      fslice="%06d" % (i)
+      for filetmp in os.listdir(args.datadir+'/plots'):
+        file=args.datadir+'/plots/'+filetmp
+        if filetmp.endswith('_r'+ fslice +'.png'):
+          idxx=int(filetmp[-10:-4])
+          os.system('ln -s '+file+' movie'+ str(idxx) +'.png')
+      os.system('ffmpeg -framerate 15 -i "movie%d.png" -pix_fmt yuv420p -c:a copy -crf 20 -r 15 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -codec:v libx264 "movie.mov"')
+      os.rename(args.datadir+'/plots/tmp/movie.mov',odir+'/'+args.outfile+'_r'+fslice+'.mov')
+      if os.path.exists("movie.mov"):
+        os.remove('movie.mov')
+  elif (args.s):
+    if (args.s > dim3):
+      fslice="%06d" % (dim3)
+      for filetmp in os.listdir(args.datadir+'/plots'):
+        file=args.datadir+'/plots/'+filetmp
+        if filetmp.endswith('_r'+ fslice +'.png'):
+          idxx=int(filetmp[-10:-4])
+          os.system('ln -s '+file+' movie'+ str(idxx) +'.png')
+      os.system('ffmpeg -framerate 15 -i "movie%d.png" -pix_fmt yuv420p -c:a copy -crf 20 -r 15 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -codec:v libx264 "movie.mov"')
+      os.rename(args.datadir+'/plots/tmp/movie.mov',odir+'/'+args.outfile+'_r'+fslice+'.mov')
+    else:
+      fslice="%06d" % (args.s)
+      for filetmp in os.listdir(args.datadir+'/plots'):
+        file=args.datadir+'/plots/'+filetmp
+        if filetmp.endswith('_r'+ fslice +'.png'):
+          idxx=int(filetmp[-10:-4])
+          os.system('ln -s '+file+' movie'+ str(idxx) +'.png')
+      os.system('ffmpeg -framerate 15 -i "movie%d.png" -pix_fmt yuv420p -c:a copy -crf 20 -r 15 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -codec:v libx264 "movie.mov"')
+      os.rename(args.datadir+'/plots/tmp/movie.mov',odir+'/'+args.outfile+'_r'+fslice+'.mov')
+  else:
+    for filetmp in os.listdir(args.datadir+'/plots'):
+      file=args.datadir+'/plots/'+filetmp
+      if filetmp.endswith('.png') and file.contains('idx'):
+        idxx=int(filetmp[-10:-4])
+        os.system('ln -s '+file+' movie'+ str(idxx) +'.png')
+    os.system('ffmpeg -framerate 15 -i "movie%d.png" -pix_fmt yuv420p -c:a copy -crf 20 -r 15 -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -codec:v libx264 "movie.mov"')
+    os.rename(args.datadir+'/plots/tmp/movie.mov',odir+'/'+args.outfile+'_r000001.mov')
+
+  for filetmp in os.listdir(args.datadir+'/plots/tmp'):
+    os.remove(filetmp)
+  os.chdir(args.datadir+'/plots')
+  os.rmdir(args.datadir+'/plots/tmp')
+
+
+
+def main():
+  ## Get input agruments:
+  args = argParsing()
+  run(args)
+
+if __name__ == '__main__':
+  main()