examples/Example3-PROC_HPLOGISITIC-SGF2021-Simple_and_Efficient_Bootstrap_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Framework for Simple and Efficient Bootstrap Validation in SAS®, with Examples\n",
    "\n",
    "\n",
    "## Example 3: PROC HPLOGISTIC &nbsp;&bullet;&nbsp; Invited Paper &nbsp;&bullet;&nbsp;  SAS Global Forum 2021\n",
    "\n",
    "This notebook contains an executable version of the example in Appendix D of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 1\n",
    "\n",
    "The code below creates macro variables encapsulating model parameters. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n",
    "\n",
    "**Note**: You may also wish to change the values of the macro variables in order to explore bootstrap validation for different models. This example is adapted from Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%let response_variable_condition = bpxsar >= 140 OR bpxdar >= 90 OR bpq050a = 1;\n",
    "%let response_variable = hyper;\n",
    "%let outcome = &response_variable.(EVENT='1');\n",
    "%let class_variables = bpq100d dmq051 dmd110;\n",
    "%let predictor_variables = lbxtc bpq100d bmxbmi ridageyr lbxtr dmq051 dmd110 indhhinc indfmpir;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 2\n",
    "\n",
    "The code below downloads the example dataset for Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename tempfile \"%sysfunc(pathname(work))/analysis_data.sas7bdat\";\n",
    "proc http\n",
    "        url='https://wwwn.cdc.gov/nchs/data/tutorials/analysis_data.sas7bdat'\n",
    "        method='get'\n",
    "        out=tempfile\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 3\n",
    "\n",
    "The code below subsets the dataset downloaded above to rows with no missing values for the predicator variables, as well as creating a response variable. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data example_dataset;\n",
    "    set analysis_data;\n",
    "    \n",
    "    * Create outcome variable;\n",
    "    if (&response_variable_condition.) then &response_variable. = 1;\n",
    "    else &response_variable. = 0;\n",
    "    \n",
    "    * Subset to observations with no missing values for outcome variable or predictor variables;\n",
    "    if nmiss(&response_variable., %sysfunc(tranwrd(&predictor_variables.,%str( ),%str(,)))) = 0;\n",
    "    \n",
    "    keep &response_variable. &predictor_variables.;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC HPLOGISTIC\n",
    "\n",
    "See page 14 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC HPLOGISTIC output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ods output Association=model_association_table(\n",
    "    keep=C\n",
    "    rename=(C=original_model_c_statistic)\n",
    ");\n",
    "proc hplogistic data=example_dataset;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables. / association;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC HPLOGISTIC (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples & Define Partition Roles\n",
    "\n",
    "See page 15 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC SURVEYSELECT output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc surveyselect\n",
    "        data=example_dataset\n",
    "        out=bootstrap_samples\n",
    "        seed=1354687\n",
    "        method=urs\n",
    "        outhits\n",
    "        rep=500\n",
    "        samprate=1\n",
    "    ;\n",
    "run;\n",
    "\n",
    "data partition_roles;\n",
    "    set bootstrap_samples;\n",
    "    by replicate;\n",
    "    test = 0;\n",
    "    output;\n",
    "    if last.replicate then do p = 1 to n;\n",
    "        set example_dataset nobs=n point=p;\n",
    "        test = 1;\n",
    "        output;\n",
    "    end;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples & Define Partition Roles (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps 3 & 4: Train and Test Models in Each Bootstrap using PROC HPLOGISTIC\n",
    "\n",
    "See pages 14-15 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. (Depending on the version of the SAS Kernel being used by Jupyter, you may see either empty output or the SAS log.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* turn off all output;\n",
    "ods graphics off;\n",
    "ods exclude all;\n",
    "ods noresults;\n",
    "\n",
    "ods output PartFitStats=bootstrap_fit(\n",
    "    where=(statistic='Area under the ROCC')\n",
    ");\n",
    "proc hplogistic data=partition_roles;\n",
    "    by replicate;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables.;\n",
    "    partition rolevar=test(train=0 test=1);\n",
    "run;\n",
    "\n",
    "* turn output back on;\n",
    "ods results;\n",
    "ods select all;\n",
    "ods graphics on;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Steps 3 & 4: Train and Test Models in Each Bootstrap using PROC HPLOGISTIC (additional optional step)\n",
    "\n",
    "If the log for the previous cell did not appear, the code below can be used to view it. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n",
    "\n",
    "**Warning**: This log entry will be particularly long since each of the 500 iterations of PROC HPLOGISITIC produces its own log entries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Estimate Optimism\n",
    "\n",
    "See page 16 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc sql;\n",
    "    create table model_optimism as\n",
    "        select \n",
    "            avg(training - testing) as optimism\n",
    "        from\n",
    "            bootstrap_fit\n",
    "    ;\n",
    "quit;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism\n",
    "\n",
    "See page 9 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC PRINT output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* assemble C-statistic for original model, optimism, and corrected C-statistic into a 1x3 table;\n",
    "data corrected_model_evaluation;\n",
    "    set model_association_table;\n",
    "    set model_optimism;\n",
    "    \n",
    "    corrected_c_statistic = original_model_c_statistic - optimism;\n",
    "\n",
    "    label\n",
    "        original_model_c_statistic = 'Naive C-Statistic'\n",
    "        optimism = 'Optimism'\n",
    "        corrected_c_statistic = 'Optimism-Corrected C-Statistic'\n",
    "    ;\n",
    "    \n",
    "    keep original_model_c_statistic optimism corrected_c_statistic;\n",
    "run;\n",
    "\n",
    "* print final results;\n",
    "proc print\n",
    "        data=corrected_model_evaluation\n",
    "        noobs\n",
    "        label\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SAS",
   "language": "sas",
   "name": "sas"
  },
  "language_info": {
   "codemirror_mode": "sas",
   "file_extension": ".sas",
   "mimetype": "text/x-sas",
   "name": "sas"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}