Experimental Design

Experimental Design Techniques in D2D

Measuring data for informative experimental conditions can reduce the necessary amount of experiments needed to inform the unknown model parameters. There are two frequentist techniques implemented into D2D which provide the necessary tools needed to plan optimal experiments. The measure which is optimized by the proposed experimental planning procedures aims at minimizing the uncertainty about a target parameter of interest.

Sampling the Likelihood Profile: arPLETrajectories

Method

The method proposed here is easy to use, computationally efficient and won the Best-Performer Award in the DREAM6-Challenge (Steiert, 2012). The experimental design procedure is described in the paper, but the general idea is the following: The profile likelihood of a parameter implies a set of parameter vectors which are compatible with the existing data. Sampling the trajectories of different possible observables for a representative subset of the previously mentioned parameter vectors provides a set of trajectories which contains information about the spread of the plausible model predictions. Measuring an observable at a time point at which the spread is large effectively constrains the possible predictions and likely also the chosen target parameter.

Implementation

The function arPLETrajectories generates the sampled trajectories. This function has two main inputs: The first input is an index vector of parameters from which the trajectories are sampled. The second input is the number of parameter vectors for which the profile is sampled.

Sampling the Space of Measurement Outcomes and Parameter Simultaneously: 2D-Profiles

Method

The method of 2D-Profiles provides a more detailed and purposeful approach to experimental design than the previous strategy. This comes at a higher computational cost than the previous approach and is therefore especially adequate if gathering data requires much resources. The 2D-Profile approach takes a more direct route to the profile likelihood of the target parameter by directly considering the improvement of the profile likelihood by a measurement. For each experimental condition, there are different possible measurement outcomes which all lead to different parameter profile likelihoods. This implies a two-dimensional profile likelihood, which contains the profile likelihood for each plausible measurement outcome. From this 2D-Profile, an expected average profile width can be calculated, which intuitively assigns an average width to each parameter profile likelihood and takes the expectation of this average profile width weighted by the plausibility of the different measurement outcomes.

In practical applications, this is the order in which you need to proceed to calculate and evaluate a 2D-Profile:

1. Find the best fit
2. Calculate parameter likelihood profiles
3. Choose target parameter based on their likelihood profiles (likely non-identifiable parameter)
4. Calculate validation profile for an experimental condition
5. Calculate 2D-Profile
6. Calculate expected average profile width

Implementation

For more information on sampling the parameter profile likelihoods, see the function ple and the wiki page Adapting the PLE Settings. The validation profile likelihood is sampled by the function VPL and described in the article Estimating Prediction uncertainty.

The main functions of the 2D-Profile Likelihood approach are the following:

• Init2DPL(npred_chi,del): This function initializes the struct ar.ple2d in which the calculation results and algorithm options are stored. The two arguments correspond to two possible ways to set the amount of measurement outomces for which the corresponding parameter profile likelihoods are sampled. The easiest way is to set npred_chi which then samples npred_chi profiles each toward the upper and lower end of plausible measurement outcomes.
• gen2d(idpar): This function samples the 2D-Profile for the parameter with index idpar. Note: The experimental condition is taken from the last validation profile calculation stored in ar.vpl, since the result of the validation profile calculation is required in order to set the range of plasuible measurement outcomes.
• autofix2dprofile: This function automatically detects and fixes errors which occurred while sampling the 2D-Profile likelihood. It is recommended to call this after any 2D-Profile calculation despite the additional computation time. If the result seems unsatisfactory, there are some magic factors in ar.ple2d.config.autofix which can be used to fix this. Note: This whole algorithm is a complex piece of code consisting of many subroutines, so finding the problem by yourself might be difficult.
• bounds2d(alphapar_max): This function calculates the parameter confidence intervals at the alphapar_max level for a representative set of measurement outcomes. There are various outputs and results can be saved into ar.ple2d.bounds. There are some additional options for this function stored in ar.ple2d.config.bounds. If one is only interested in the expected average width, this function does not need to be called separately, but it is has its use e.g. if the bounds are to be calculated for plotting.
• score = score2d(alphapar_step,alphapar_max): This function calculates the expected average profile width from the 2D-Profile (score). There are additional outputs if one is interested in more detailed results. The input alphapar_step is a scalar value which specifies the equidistant parameter confidence levels which are used for averaging the profile width and alphapar_max corresponds to the maximum alpha to which parameter confidence intervals are averaged.
• Plot2DPL(plotmode): Generic plot function which either plots the 2d-profile as a scatter plot of all the points for which the 2d-profile value was sampled (plotmode =1) or as a contour plot (plotmode = 2). The latter option is usually more elegant if the output makes any sense, while the former option is more robust and should always provide insight. If bounds2d is called before the contour plot is generated, the parameter confidence bounds at the specified confidence levels are visible which allows for an easy visualization of the information content of the plotted 2D-Profile.

Example

Example code to calculate and evaluate a 2D-Profile for the ABC-model is provided here.