Merge pull request #195 from stan-dev/version-2.5.0

fix for CRAN and update NEWS

DESCRIPTION

@@ -1,8 +1,8 @@
 Package: loo
 Type: Package
 Title: Efficient Leave-One-Out Cross-Validation and WAIC for Bayesian Models
-Version: 2.4.1.9000
-Date: 2020-12-07
+Version: 2.5.0
+Date: 2022-03-02
 Authors@R: c(person("Aki", "Vehtari", email = "Aki.Vehtari@aalto.fi", role = c("aut")),
              person("Jonah", "Gabry", email = "jsg2201@columbia.edu", role = c("cre", "aut")),
              person("Mans", "Magnusson", role = c("aut")),
@@ -51,5 +51,5 @@ Suggests:
 VignetteBuilder: knitr
 Encoding: UTF-8
 SystemRequirements: pandoc (>= 1.12.3), pandoc-citeproc
-RoxygenNote: 7.1.1
+RoxygenNote: 7.1.2
 Roxygen: list(markdown = TRUE)

NEWS.md

@@ -1,10 +1,13 @@
-# Items for next release go here
+# loo 2.5.0
+
+### Improvements
+
+* New [Frequently Asked Questions page](https://mc-stan.org/loo/articles/online-only/faq.html) on the package website. (#143)
 
 * Speed improvement from simplifying the normalization when fitting the 
 generalized Pareto distribution. (#187, @sethaxen)
 
-* Fixed bug where attribute storing normalizing constant of PSIS weights wasn't
-updated when using moment matching. (#167, @fweber144)
+* Added parallel likelihood computation to speedup `loo_subsample()` when using posterior approximations. (#171, @kdubovikov)
 
 * Switch unit tests from Travis to GitHub Actions. (#164)
 
@@ -13,6 +16,9 @@ updated when using moment matching. (#167, @fweber144)
 * Fixed a bug causing the normalizing constant of the PSIS (log) weights not 
 to get updated when performing moment matching with `save_psis = TRUE` (#166, @fweber144).
 
+* Fixed bug where the attribute storing normalizing constant of PSIS weights
+wasn't updated when using moment matching. (#167, @fweber144)
+
 # loo 2.4.1
 
 * Fixed issue reported by CRAN where one of the vignettes errored on an M1 Mac

R/diagnostics.R

@@ -80,7 +80,10 @@
 #'  **over-optimistic when the estimate of \eqn{k} is greater than 0.7**.
 #' }
 #'
-#' @seealso [psis()] for the implementation of the PSIS algorithm.
+#' @seealso
+#'  * [psis()] for the implementation of the PSIS algorithm.
+#'  * The [FAQ page](https://mc-stan.org/loo/articles/online-only/faq.html) on
+#'    the __loo__ website for answers to frequently asked questions.
 #'
 #' @template loo-and-psis-references
 #'

R/loo-glossary.R

@@ -5,7 +5,12 @@
 #' @template loo-and-psis-references
 #' @template bayesvis-reference
 #'
-#' @description Note: VGG2017 refers to Vehtari, Gelman, and Gabry (2017). See
+#' @description
+#'   The pages provides definitions to key terms. Also see the
+#'   [FAQ page](https://mc-stan.org/loo/articles/online-only/faq.html) on
+#'   the __loo__ website for answers to frequently asked questions.
+#'
+#'   Note: VGG2017 refers to Vehtari, Gelman, and Gabry (2017). See
 #'   **References**, below.
 #'
 #' @section ELPD and `elpd_loo`:

R/loo-package.R

@@ -8,7 +8,7 @@
 #'
 #' @description
 #' \if{html}{
-#'   \figure{stanlogo.png}{options: width="50px" alt="mc-stan.org"}
+#'   \figure{stanlogo.png}{options: width="50" alt="mc-stan.org"}
 #' }
 #' *Stan Development Team*
 #'

R/loo.R

@@ -17,16 +17,16 @@
 #'   with MCMC. If MCMC draws are used and `r_eff` is not provided then
 #'   the reported PSIS effective sample sizes and Monte Carlo error estimates
 #'   will be over-optimistic. If the posterior draws are independent then
-#'   `r_eff=1` and can be omitted. See the [relative_eff()]
-#'   helper functions for computing `r_eff`.
-#'
+#'   `r_eff=1` and can be omitted. The warning message thrown when `r_eff` is
+#'   not specified can be disabled by setting `r_eff` to `NA`. See the
+#'   [relative_eff()] helper functions for computing `r_eff`.
 #' @param save_psis Should the `"psis"` object created internally by `loo()` be
 #'   saved in the returned object? The `loo()` function calls [psis()]
 #'   internally but by default discards the (potentially large) `"psis"` object
 #'   after using it to compute the LOO-CV summaries. Setting `save_psis=TRUE`
-#'   will add a `psis_object` component to the list returned by `loo`. Currently
-#'   this is only needed if you plan to use the [E_loo()] function to compute
-#'   weighted expectations after running `loo`.
+#'   will add a `psis_object` component to the list returned by `loo`.
+#'   Currently this is only needed if you plan to use the [E_loo()] function to
+#'   compute weighted expectations after running `loo`.
 #' @template cores
 #' @template is_method
 #'
@@ -47,32 +47,31 @@
 #'   `c("psis_loo", "loo")` and components:
 #' \describe{
 #'  \item{`estimates`}{
-#'   A matrix with two columns (`Estimate`, `SE`) and three rows
-#'   (`elpd_loo`, `p_loo`, `looic`). This
-#'   contains point estimates and standard errors of the expected log pointwise
-#'   predictive density (`elpd_loo`), the effective number of parameters
-#'   (`p_loo`) and the LOO information criterion `looic` (which is
-#'   just `-2 * elpd_loo`, i.e., converted to deviance scale).
+#'   A matrix with two columns (`Estimate`, `SE`) and three rows (`elpd_loo`,
+#'   `p_loo`, `looic`). This contains point estimates and standard errors of the
+#'   expected log pointwise predictive density ([`elpd_loo`][loo-glossary]), the
+#'   effective number of parameters ([`p_loo`][loo-glossary]) and the LOO
+#'   information criterion `looic` (which is just `-2 * elpd_loo`, i.e.,
+#'   converted to deviance scale).
 #'  }
 #'
 #'  \item{`pointwise`}{
 #'   A matrix with five columns (and number of rows equal to the number of
 #'   observations) containing the pointwise contributions of the measures
 #'   (`elpd_loo`, `mcse_elpd_loo`, `p_loo`, `looic`, `influence_pareto_k`).
-#'   in addition to the three measures in \code{estimates}, we also report
-#'   pointwise values of the Monte Carlo standard error of \code{elpd_loo}
-#'   (\code{mcse_elpd_loo}), and statistics describing the influence of
-#'   each observation on the posterior distribution (\code{influence_pareto_k}).
+#'   in addition to the three measures in `estimates`, we also report
+#'   pointwise values of the Monte Carlo standard error of [`elpd_loo`][loo-glossary]
+#'   ([`mcse_elpd_loo`][loo-glossary]), and statistics describing the influence of
+#'   each observation on the posterior distribution (`influence_pareto_k`).
 #'   These are the estimates of the shape parameter \eqn{k} of the
 #'   generalized Pareto fit to the importance ratios for each leave-one-out
-#'   distribution. See the [pareto-k-diagnostic] page for details.
+#'   distribution (see the [pareto-k-diagnostic] page for details).
 #'  }
 #'
-#'
 #'  \item{`diagnostics`}{
 #'  A named list containing two vectors:
 #'    * `pareto_k`: Importance sampling reliability diagnostics. By default,
-#'      these are equal to the \code{influence_pareto_k} in \code{pointwise}.
+#'      these are equal to the `influence_pareto_k` in `pointwise`.
 #'      Some algorithms can improve importance sampling reliability and
 #'      modify these diagnostics. See the [pareto-k-diagnostic] page for details.
 #'    * `n_eff`: PSIS effective sample size estimates.
@@ -87,8 +86,10 @@
 #' }
 #'
 #' @seealso
-#'  * The **loo** package [vignettes](https://mc-stan.org/loo/articles/index.html)
+#'  * The __loo__ package [vignettes](https://mc-stan.org/loo/articles/index.html)
 #'    for demonstrations.
+#'  * The [FAQ page](https://mc-stan.org/loo/articles/online-only/faq.html) on
+#'    the __loo__ website for answers to frequently asked questions.
 #'  * [psis()] for the underlying Pareto Smoothed Importance Sampling (PSIS)
 #'    procedure used in the LOO-CV approximation.
 #'  * [pareto-k-diagnostic] for convenience functions for looking at diagnostics.

R/loo_compare.R

@@ -17,31 +17,34 @@
 #'
 #' @details
 #'   When comparing two fitted models, we can estimate the difference in their
-#'   expected predictive accuracy by the difference in `elpd_loo` or
-#'   `elpd_waic` (or multiplied by \eqn{-2}, if desired, to be on the
-#'   deviance scale).
+#'   expected predictive accuracy by the difference in
+#'   [`elpd_loo`][loo-glossary] or `elpd_waic` (or multiplied by \eqn{-2}, if
+#'   desired, to be on the deviance scale).
 #'
-#'   When using `loo_compare()`, the returned matrix will have one row per
-#'   model and several columns of estimates. The values in the `elpd_diff`
-#'   and `se_diff` columns of the returned matrix are computed by making
-#'   pairwise comparisons between each model and the model with the largest ELPD
-#'   (the model in the first row). For this reason the `elpd_diff` column
-#'   will always have the value `0` in the first row (i.e., the difference
-#'   between the preferred model and itself) and negative values in subsequent
-#'   rows for the remaining models.
+#'   When using `loo_compare()`, the returned matrix will have one row per model
+#'   and several columns of estimates. The values in the
+#'   [`elpd_diff`][loo-glossary] and [`se_diff`][loo-glossary] columns of the
+#'   returned matrix are computed by making pairwise comparisons between each
+#'   model and the model with the largest ELPD (the model in the first row). For
+#'   this reason the `elpd_diff` column will always have the value `0` in the
+#'   first row (i.e., the difference between the preferred model and itself) and
+#'   negative values in subsequent rows for the remaining models.
 #'
-#'   To compute the standard error of the difference in ELPD --- which should
-#'   not be expected to equal the difference of the standard errors --- we use a
-#'   paired estimate to take advantage of the fact that the same set of \eqn{N}
-#'   data points was used to fit both models. These calculations should be most
-#'   useful when \eqn{N} is large, because then non-normality of the
-#'   distribution is not such an issue when estimating the uncertainty in these
-#'   sums. These standard errors, for all their flaws, should give a better
-#'   sense of uncertainty than what is obtained using the current standard
-#'   approach of comparing differences of deviances to a Chi-squared
+#'   To compute the standard error of the difference in [ELPD][loo-glossary] ---
+#'   which should not be expected to equal the difference of the standard errors
+#'   --- we use a paired estimate to take advantage of the fact that the same
+#'   set of \eqn{N} data points was used to fit both models. These calculations
+#'   should be most useful when \eqn{N} is large, because then non-normality of
+#'   the distribution is not such an issue when estimating the uncertainty in
+#'   these sums. These standard errors, for all their flaws, should give a
+#'   better sense of uncertainty than what is obtained using the current
+#'   standard approach of comparing differences of deviances to a Chi-squared
 #'   distribution, a practice derived for Gaussian linear models or
 #'   asymptotically, and which only applies to nested models in any case.
 #'
+#' @seealso
+#' * The [FAQ page](https://mc-stan.org/loo/articles/online-only/faq.html) on
+#'   the __loo__ website for answers to frequently asked questions.
 #' @template loo-and-psis-references
 #'
 #' @examples

R/psis.R

@@ -68,6 +68,10 @@
 #' @seealso
 #' * [loo()] for approximate LOO-CV using PSIS.
 #' * [pareto-k-diagnostic] for PSIS diagnostics.
+#' * The __loo__ package [vignettes](https://mc-stan.org/loo/articles/index.html)
+#'   for demonstrations.
+#' * The [FAQ page](https://mc-stan.org/loo/articles/online-only/faq.html) on
+#'   the __loo__ website for answers to frequently asked questions.
 #'
 #' @template loo-and-psis-references
 #'

man-roxygen/moment-matching-references.R

@@ -1,5 +1,4 @@
 #' @references
-#' Paananen, T., Piironen, J., Buerkner, P.-C., Vehtari, A. (2020).
-#' Implicitly Adaptive Importance Sampling.
-#' [preprint arXiv:1906.08850](https://arxiv.org/abs/1906.08850)
-#'
+#' Paananen, T., Piironen, J., Buerkner, P.-C., Vehtari, A. (2021).
+#' Implicitly adaptive importance sampling. *Statistics and Computing*, 31, 16.
+#' doi:10.1007/s11222-020-09982-2. arXiv preprint arXiv:1906.08850.