diff --git a/docs/Users_Guide/glossary.rst b/docs/Users_Guide/glossary.rst index 4ed26829b..94b11cc28 100644 --- a/docs/Users_Guide/glossary.rst +++ b/docs/Users_Guide/glossary.rst @@ -12990,3 +12990,280 @@ METplus Configuration Glossary Specify the value for 'obtype_as_group_val_flag' in the MET configuration file for EnsembleStat. | *Used by:* EnsembleStat + + ALLOW_MISSING_INPUTS + If True, report a warning instead of an error if a wrapper cannot find all + of its required input files. Only report an error if the number of runs + that successfully found input files does not meet the threshold defined + with :term:`INPUT_THRESH`. + Wrapper-specific versions of this variable are supported for many of the + wrappers. Defaults to False. + + | *Used by:* Many + + INPUT_THRESH + A decimal number between 0 and 1 that defines the threshold of successful + run times where all of the required input files were found. + If INPUT_THRESH is set to 0.5 and 3 out of the 4 run times (75% or 0.75) + successfully found the required input files, no errors will be reported. + However, if INPUT_THRESH is set to 0.5 and only 1 out of 4 run times + (25% or 0.25) found the required input files, an error will be reported + at the end of the run. The threshold is inclusive, so 2 out of 4 successful + run times (50% or 0.5) will not report an error. + This is only used if :term:`ALLOW_MISSING_INPUTS` is True. + Wrapper-specific versions of this variable are supported for many of the + wrappers. + + | *Used by:* Many + + ASCII2NC_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for ASCII2NC only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* ASCII2NC + + ASCII2NC_INPUT_THRESH + Defines input threshold for ASCII2NC only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* ASCII2NC + + ENSEMBLE_STAT_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for EnsembleStat only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* EnsembleStat + + ENSEMBLE_STAT_INPUT_THRESH + Defines input threshold for EnsembleStat only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* EnsembleStat + + GEN_ENS_PROD_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for GenEnsProd only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* GenEnsProd + + GEN_ENS_PROD_INPUT_THRESH + Defines input threshold for GenEnsProd only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* GenEnsProd + + GEN_VX_MASK_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for GenVxMask only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* GenVxMask + + GEN_VX_MASK_INPUT_THRESH + Defines input threshold for GenVxMask only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* GenVxMask + + GRID_DIAG_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for GridDiag only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* GridDiag + + GRID_DIAG_INPUT_THRESH + Defines input threshold for GridDiag only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* GridDiag + + GRID_STAT_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for GridStat only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* GridStat + + GRID_STAT_INPUT_THRESH + Defines input threshold for GridStat only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* GridStat + + IODA2NC_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for IODA2NC only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* IODA2NC + + IODA2NC_INPUT_THRESH + Defines input threshold for IODA2NC only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* IODA2NC + + MADIS2NC_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for MADIS2NC only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* MADIS2NC + + MADIS2NC_INPUT_THRESH + Defines input threshold for MADIS2NC only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* MADIS2NC + + MODE_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for MODE only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* MODE + + MODE_INPUT_THRESH + Defines input threshold for MODE only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* MODE + + MTD_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for MTD only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* MTD + + MTD_INPUT_THRESH + Defines input threshold for MTD only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* MTD + + PB2NC_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for PB2NC only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* PB2NC + + PB2NC_INPUT_THRESH + Defines input threshold for PB2NC only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* PB2NC + + PCP_COMBINE_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for PCPCombine only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* PCPCombine + + PCP_COMBINE_INPUT_THRESH + Defines input threshold for PCPCombine only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* PCPCombine + + PLOT_DATA_PLANE_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for PlotDataPlane only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* PlotDataPlane + + PLOT_DATA_PLANE_INPUT_THRESH + Defines input threshold for PlotDataPlane only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* PlotDataPlane + + PLOT_POINT_OBS_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for PlotPointObs only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* PlotPointObs + + PLOT_POINT_OBS_INPUT_THRESH + Defines input threshold for PlotPointObs only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* PlotPointObs + + POINT2GRID_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for Point2Grid only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* Point2Grid + + POINT2GRID_INPUT_THRESH + Defines input threshold for Point2Grid only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* Point2Grid + + POINT_STAT_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for PointStat only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* PointStat + + POINT_STAT_INPUT_THRESH + Defines input threshold for PointStat only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* PointStat + + REGRID_DATA_PLANE_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for RegridDataPlane only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* RegridDataPlane + + REGRID_DATA_PLANE_INPUT_THRESH + Defines input threshold for RegridDataPlane only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* RegridDataPlane + + SERIES_ANALYSIS_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for SeriesAnalysis only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* SeriesAnalysis + + SERIES_ANALYSIS_INPUT_THRESH + Defines input threshold for SeriesAnalysis only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* SeriesAnalysis + + TC_DIAG_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for TCDiag only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* TCDiag + + TC_DIAG_INPUT_THRESH + Defines input threshold for TCDiag only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* TCDiag + + TC_GEN_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for TCGen only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* TCGen + + TC_GEN_INPUT_THRESH + Defines input threshold for TCGen only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* TCGen + + WAVELET_STAT_ALLOW_MISSING_INPUTS + Activates allow missing inputs logic for WaveletStat only. + See :term:`ALLOW_MISSING_INPUTS` for details. + + | *Used by:* WaveletStat + + WAVELET_STAT_INPUT_THRESH + Defines input threshold for WaveletStat only. + See :term:`INPUT_THRESH` for details. + + | *Used by:* WaveletStat diff --git a/docs/Users_Guide/systemconfiguration.rst b/docs/Users_Guide/systemconfiguration.rst index cdffb1ffc..ac84ffb02 100644 --- a/docs/Users_Guide/systemconfiguration.rst +++ b/docs/Users_Guide/systemconfiguration.rst @@ -2138,6 +2138,96 @@ can be simplified as:: INPUT_TEMPLATE = ensbegin_end_incr(1,8,1,2).nc +.. _allow-missing-inputs: + +Allow Missing Inputs +-------------------- + +When any of the required input files for a given METplus run time are not found, +an error is reported. In result, the entire METplus run fails. +In some cases, users may expect a certain number of inputs to be unavailable +and do not want the entire run to fail when this happens. + +The :term:`ALLOW_MISSING_INPUTS` config variable can be set to **True** to +report a warning when required inputs are not found for a given run time. +An error at the end of the METplus run will only be reported if the number +of successful runs does not meet the value defined by :term:`INPUT_THRESH`. +The value of :term:`INPUT_THRESH` should be a decimal number between 0 and 1. +The default value is 0.0, so any missing input files in a run will still report +an error unless this value is changed. + +The threshold is compared to the results of each item in the +:ref:`Process_List`, so each wrapper listed in the **PROCESS_LIST** must meet +the threshold to prevent an error. + +There are wrapper-specific versions of both :term:`ALLOW_MISSING_INPUTS` and +:term:`INPUT_THRESH` for most of the wrappers, +e.g. :term:`GRID_STAT_ALLOW_MISSING_INPUTS` and :term:`GRID_STAT_INPUT_THRESH`. +Refer to the :ref:`python_wrappers` chapter or the :ref:`METplus_glossary` +to see which variables are supported. + +**Example 1**:: + + [config] + PROCESS_LIST = RegridDataPlane, GridStat + VALID_TIME_FMT = %Y%m%d%H + VALID_BEG = 2024020301 + VALID_BEG = 2024020310 + VALID_INCREMENT = 1H + LEAD_SEQ = 0 + + ALLOW_MISSING_INPUTS = True + INPUT_THRESH = 0.6 + +In this example, 10 valid times will be run, so there will be 10 calls to +RegridDataPlane and 10 calls to GridStat. The input threshold is set +to 60%, so if 6 or more of the RegridDataPlane runs *and* +6 or more of the GridStat runs successfully find all of the required files, +an error will not be reported. If 5 or fewer runs for either wrapper succeed, +then an error will be reported. + +**Example 2**:: + + + [config] + PROCESS_LIST = RegridDataPlane, GridStat + VALID_TIME_FMT = %Y%m%d%H + VALID_BEG = 2024020301 + VALID_BEG = 2024020310 + VALID_INCREMENT = 1H + LEAD_SEQ = 0 + + GRID_STAT_ALLOW_MISSING_INPUTS = True + INPUT_THRESH = 0.6 + +In this case, only GridStat wrapper will allow missing inputs. +At least 60% of the GridStat runs +must successfully find the required input files to prevent an error. +Any missing inputs for RegridDataPlane will result in an error. + +**Example 3**:: + + + [config] + PROCESS_LIST = RegridDataPlane, GridStat + VALID_TIME_FMT = %Y%m%d%H + VALID_BEG = 2024020301 + VALID_BEG = 2024020310 + VALID_INCREMENT = 1H + LEAD_SEQ = 0 + + ALLOW_MISSING_INPUTS = True + REGRID_DATA_PLANE_INPUT_THRESH = 0.9 + GRID_STAT_INPUT_THRESH = 0.6 + +In this case, both GridStat and RegridDataPlane wrappers allow missing inputs, +but the threshold to prevent an error differs between wrappers. +At least 90% of the RegridDataPlane runs +must successfully find the required input files to prevent an error. +At least 60% of the GridStat runs +must successfully find the required input files to prevent an error. +Any missing inputs for RegridDataPlane will result in an error. + .. _metplus-control-met: How METplus controls MET configuration variables diff --git a/docs/Users_Guide/wrappers.rst b/docs/Users_Guide/wrappers.rst index 5ca93e9dd..dc3106fd0 100644 --- a/docs/Users_Guide/wrappers.rst +++ b/docs/Users_Guide/wrappers.rst @@ -64,6 +64,9 @@ METplus Configuration | :term:`ASCII2NC_INC_VALID_TIMES` | :term:`ASCII2NC_SKIP_INIT_TIMES` | :term:`ASCII2NC_INC_INIT_TIMES` +| :term:`ASCII2NC_ALLOW_MISSING_INPUTS` +| :term:`ASCII2NC_INPUT_THRESH` + | .. _ascii2nc-met-conf: @@ -380,6 +383,8 @@ METplus Configuration | :term:`ENSEMBLE_STAT_INC_VALID_TIMES` | :term:`ENSEMBLE_STAT_SKIP_INIT_TIMES` | :term:`ENSEMBLE_STAT_INC_INIT_TIMES` +| :term:`ENSEMBLE_STAT_ALLOW_MISSING_INPUTS` +| :term:`ENSEMBLE_STAT_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -1352,6 +1357,8 @@ METplus Configuration | :term:`GEN_ENS_PROD_INC_VALID_TIMES` | :term:`GEN_ENS_PROD_SKIP_INIT_TIMES` | :term:`GEN_ENS_PROD_INC_INIT_TIMES` +| :term:`GEN_ENS_PROD_ALLOW_MISSING_INPUTS` +| :term:`GEN_ENS_PROD_INPUT_THRESH` .. _gen-ens-prod-met-conf: @@ -1746,6 +1753,8 @@ Configuration | :term:`GEN_VX_MASK_INC_VALID_TIMES` | :term:`GEN_VX_MASK_SKIP_INIT_TIMES` | :term:`GEN_VX_MASK_INC_INIT_TIMES` +| :term:`GEN_VX_MASK_ALLOW_MISSING_INPUTS` +| :term:`GEN_VX_MASK_INPUT_THRESH` | .. _gfdl_tracker_wrapper: @@ -3054,6 +3063,8 @@ METplus Configuration | :term:`GRID_DIAG_INC_VALID_TIMES` | :term:`GRID_DIAG_SKIP_INIT_TIMES` | :term:`GRID_DIAG_INC_INIT_TIMES` +| :term:`GRID_DIAG_ALLOW_MISSING_INPUTS` +| :term:`GRID_DIAG_INPUT_THRESH` | .. _grid-diag-met-conf: @@ -3399,6 +3410,8 @@ METplus Configuration | :term:`GRID_STAT_UGRID_COORDINATES_FILE` | :term:`GRID_STAT_UGRID_CONFIG_FILE` | :term:`GRID_STAT_TIME_OFFSET_WARNING` +| :term:`GRID_STAT_ALLOW_MISSING_INPUTS` +| :term:`GRID_STAT_INPUT_THRESH` | .. warning:: **DEPRECATED** @@ -4169,6 +4182,8 @@ METplus Configuration | :term:`IODA2NC_INC_VALID_TIMES` | :term:`IODA2NC_SKIP_INIT_TIMES` | :term:`IODA2NC_INC_INIT_TIMES` +| :term:`IODA2NC_ALLOW_MISSING_INPUTS` +| :term:`IODA2NC_INPUT_THRESH` .. _ioda2nc-met-conf: @@ -4451,6 +4466,8 @@ METplus Configuration | :term:`MADIS2NC_INC_VALID_TIMES` | :term:`MADIS2NC_SKIP_INIT_TIMES` | :term:`MADIS2NC_INC_INIT_TIMES` +| :term:`MADIS2NC_ALLOW_MISSING_INPUTS` +| :term:`MADIS2NC_INPUT_THRESH` | .. _madis2nc-met-conf: @@ -4878,6 +4895,8 @@ METplus Configuration | :term:`MODE_SKIP_INIT_TIMES` | :term:`MODE_INC_INIT_TIMES` | :term:`MODE_TIME_OFFSET_WARNING` +| :term:`MODE_ALLOW_MISSING_INPUTS` +| :term:`MODE_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -5638,6 +5657,8 @@ METplus Configuration | :term:`MTD_INC_VALID_TIMES` | :term:`MTD_SKIP_INIT_TIMES` | :term:`MTD_INC_INIT_TIMES` +| :term:`MTD_ALLOW_MISSING_INPUTS` +| :term:`MTD_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -5930,6 +5951,8 @@ METplus Configuration | :term:`PB2NC_INC_VALID_TIMES` | :term:`PB2NC_SKIP_INIT_TIMES` | :term:`PB2NC_INC_INIT_TIMES` +| :term:`PB2NC_ALLOW_MISSING_INPUTS` +| :term:`PB2NC_INPUT_THRESH` .. warning:: **DEPRECATED:** @@ -6275,6 +6298,8 @@ METplus Configuration | :term:`FCST_PCP_COMBINE_VLD_THRESH` | :term:`OBS_PCP_COMBINE_INPUT_THRESH` | :term:`OBS_PCP_COMBINE_VLD_THRESH` +| :term:`PCP_COMBINE_ALLOW_MISSING_INPUTS` +| :term:`PCP_COMBINE_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -6339,6 +6364,8 @@ Configuration | :term:`PLOT_DATA_PLANE_INC_VALID_TIMES` | :term:`PLOT_DATA_PLANE_SKIP_INIT_TIMES` | :term:`PLOT_DATA_PLANE_INC_INIT_TIMES` +| :term:`PLOT_DATA_PLANE_ALLOW_MISSING_INPUTS` +| :term:`PLOT_DATA_PLANE_INPUT_THRESH` .. _plot_point_obs_wrapper: @@ -6406,6 +6433,8 @@ Configuration | :term:`PLOT_POINT_OBS_INC_VALID_TIMES` | :term:`PLOT_POINT_OBS_SKIP_INIT_TIMES` | :term:`PLOT_POINT_OBS_INC_INIT_TIMES` +| :term:`PLOT_POINT_OBS_ALLOW_MISSING_INPUTS` +| :term:`PLOT_POINT_OBS_INPUT_THRESH` .. _plot-point-obs-met-conf: @@ -6793,6 +6822,8 @@ METplus Configuration | :term:`POINT2GRID_OBS_QUALITY_INC` | :term:`POINT2GRID_OBS_QUALITY_EXC` | :term:`POINT2GRID_MET_CONFIG_OVERRIDES` +| :term:`POINT2GRID_ALLOW_MISSING_INPUTS` +| :term:`POINT2GRID_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -7125,6 +7156,8 @@ Configuration | :term:`POINT_STAT_UGRID_COORDINATES_FILE` | :term:`POINT_STAT_UGRID_CONFIG_FILE` | :term:`POINT_STAT_POINT_WEIGHT_FLAG` +| :term:`POINT_STAT_ALLOW_MISSING_INPUTS` +| :term:`POINT_STAT_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -7926,6 +7959,8 @@ METplus Configuration | :term:`REGRID_DATA_PLANE_INC_VALID_TIMES` | :term:`REGRID_DATA_PLANE_SKIP_INIT_TIMES` | :term:`REGRID_DATA_PLANE_INC_INIT_TIMES` +| :term:`REGRID_DATA_PLANE_ALLOW_MISSING_INPUTS` +| :term:`REGRID_DATA_PLANE_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -8107,6 +8142,8 @@ METplus Configuration | :term:`SERIES_ANALYSIS_SKIP_INIT_TIMES` | :term:`SERIES_ANALYSIS_INC_INIT_TIMES` | :term:`SERIES_ANALYSIS_TIME_OFFSET_WARNING` +| :term:`SERIES_ANALYSIS_ALLOW_MISSING_INPUTS` +| :term:`SERIES_ANALYSIS_INPUT_THRESH` | .. warning:: **DEPRECATED:** @@ -9442,6 +9479,8 @@ METplus Configuration | :term:`TC_DIAG_INC_VALID_TIMES` | :term:`TC_DIAG_SKIP_INIT_TIMES` | :term:`TC_DIAG_INC_INIT_TIMES` +| :term:`TC_DIAG_ALLOW_MISSING_INPUTS` +| :term:`TC_DIAG_INPUT_THRESH` | .. _tc-diag-met-conf: @@ -9974,6 +10013,8 @@ METplus Configuration | :term:`TC_GEN_DLAND_FILE` | :term:`TC_GEN_BASIN_FILE` | :term:`TC_GEN_NC_PAIRS_GRID` +| :term:`TC_GEN_ALLOW_MISSING_INPUTS` +| :term:`TC_GEN_INPUT_THRESH` .. warning:: **DEPRECATED:** @@ -12147,6 +12188,8 @@ METplus Configuration | :term:`WAVELET_STAT_WVLT_PLOT_PLOT_MAX` | :term:`WAVELET_STAT_OUTPUT_PREFIX` | :term:`WAVELET_STAT_TIME_OFFSET_WARNING` +| :term:`WAVELET_STAT_ALLOW_MISSING_INPUTS` +| :term:`WAVELET_STAT_INPUT_THRESH` .. _wavelet-stat-met-conf: